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Foreword 


The purpose of this Symposium, as suggested by its title, was to take ‘‘a hard look’’ at the procedures 
involved in the planning, evaluation and application of survey research. While this serves as a perfectly 
accurate description of our aims, it fails to impart the underlying rationale and feelings beneath the planning 
and development of this programme. This deserves a moment’s mention. 


The Latin word ‘‘circare’’ - to go around, to explore - implies circularity, and is the root of ‘‘search’’ and 
“research’’. While this fact may not be familiar to all, the process most certainly is. We, as researchers, earn 
our living, daily, by ‘‘going around’, by searching, by exploring the questions, issues and problems before us. 
In this Symposium, we examined survey research, that is, we ‘‘critically looked within’’. This in the hope that 
the concepts discussed, the ideas shared, the dialogues exchanged could but only strengthen the discipline 
at the centre of focus, and, perhaps, also those who participated. 


Drawing from T.S. Elliot's quatrain may express our intent better still. 


And the end of all our exploring 
Will be to arrive where we started 
And know the place for the first time. 


Beyond the possibilities of critical introspection, the Ottawa Symposium possessed yet a further dimension. 
One which only a joint venture of this nature could create. The Symposium offered individuals from the public 
and private sectors the chance to experience parallactic views: to observe subjects of interest from different 
perspectives. It was hoped that this opportunity, coupled with the diverse backgrounds of participants, would 
lead to a unique cross-fertilization of ideas and viewpoints. Judging from the reactions to this Symposium, we 
believe we have had a good measure of success. 


Yet regardless of how well intentioned, none of our goals would ever have been realized but for the excellent 
calibre of papers and presentations. Our heartfelt thanks go out to all the speakers for their valued 
contributions. 


Many thanks are also due to the Chairpeople for their efforts and the deligates who participated to make this 
event so well worthwhile. 


Finally, our view of the future echoes T.S. Elliot again. His quatrain begins: 


We shall not cease from exploration. 


Ow hier 


D.W. (Ted) Keller 
Organizing Committee 


Organizing Committee: 
Statistics Canada: Merv McCamon, Ed Wilhelm 


P.M.R.S.: D.W. (Ted) Keller, (D’Arcy-MacManus & Masius, Ltd.) 
Alastair Lynn, (Consumer Contact Ltd.) 
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AN OFFICIAL WELCOME TO THE SYMPOSIUM 
FROM THE PRESIDENT OF THE PROFESSIONAL 
MARKETING RESEARCH SOCIETY 


Philip H. Maynard, 
President of C.A. Research Ltd., 
Toronto, Ontario 


Ladies and Gentlemen, 


It gives me great pleasure to welcome you to this symposium, 
jointly sponsored by the P.M.R.S. and Statistics Canada. 


| am particularly pleased that we are undertaking this joint 
symposium, the second in what | hope will be an ongoing 
series. 


| think that as a profession and as a professional body, we are 
badly in need of a cross fertilisation of ideas, which this kind of 
symposium should provide. It should broaden our horizons or at 
the very least help us to understand each other better. 


| am very much in favour of joint symposiums or seminars, not 
only with Statistics Canada or even other research bodies, but 
also with marketing groups, etc. - the AMA is an obvious 
example. This is a theme, | will return to later in this speech. 


Some Comments on the Theme 


In the limited time available, | can do no more than raise a few 
important issues, issues that face us all, which other speakers 
may address and even provide solutions to. 


If not, then we will have to continue the debate at some other 
time. 


The theme of the symposium is: 


“A Critical Look at Survey Research from Planning to 
Evaluation’. 


In other words, as Ted Keller has described it, it’s a ‘“‘back to 
basics’’ theme. 


* This is an important theme at the best of times, but especially 
now, when the economy has been through, and is still 
suffering from the effects of, the worst recession since the 
1930's. 


A revolution in terms of data collection and analysis may be 
on the doorstep, but let’s re-examine the basics before 
running on beyond ourselves. 


The recession has not left the research industry totally 
unscathed, although certain companies and areas have 
prospered. 


In addition, there is now ‘light at the end of the tunnel’, 
interest rates are coming down and a period of slow recovery 
may be in prospect. 


So what better time to re-examine the basics, as a base for 
future, albeit cautious, expansion. 


* Ottawa is also an appropriate location for a symposium at 
this time, in view of the fundamental shift in the allocation of 
resources, from the private to the public sector. 


This is probably a long standing and irreversible trend, 
although it seems seems to have accelerated somewhat, 
recently. | am talking both generally and specifically about 
the advertising and marketing research areas of industry. 


There may be some positive aspects for marketing research, 
associated with the recession, apart from making us all leaner 
and hungrier. 


An important by-product of the recession may be greater 
acceptance and usage of marketing research, for two reasons: 


1. The private sector has suffered a severe ‘‘psychological 
blow’’, which means that they will be less willing to take 
risks and ‘‘fly by the seat of their pants’’, than they were in 
the past. 


This may be only a short term change, but the likelihood is 
of a more long term change in the decision making 
processes. 


For a new generation of middle to senior management, this 
is their first taste of what the 1930’s may have been like. 


Some of the ‘‘traditional virtues’’, long scorned by the 
welfare state generations, now appear to have relevance. 


2. The competitive environment, being that much tougher, 
demands more finely tuned marketing strategies, in order 
for the manufacturer to succeed or at least survive in the 
market place. 


It may be tempting, at a time such as this, to retrench, to 
batten down the shutters or merely to contemplate one’s navel. 
In the face of adversity, many of us turn inward in our 
thinking. 


No, now is the time for positive, if somewhat careful, thinking 
and action. Positive austerity is a phrase, which | think best 
sums up what should be our approach to the current situation. 


| hope that the conference will reflect his positive approach, 
rather than the more negative one mentioned earlier. We have 
work to do and what better way to start, than with a thorough 
examination of our basic methods. 


Re-examining the basics may not be the most exciting theme, 
but it is surely the most important. 


* New techniques may be very exciting, but many are dis- 
carded after a short period of time or at best become 
peripheral to the mainstream of marketing research. 


* The basic core of marketing research methods seem to 
continue, to stand the test of time. They are ‘tried and 
tested’, although we should not close our minds to outside 
influences, to new techniques and technology. 


A Critical Look 


The key word in the theme is critical. 


Let's take a critical look at what we do, but there’s surely no 
point in being critical for the sake of it. It’s too easy to do and 
destructive criticism serves no useful purpose; indeed it can 
destroy quite worthwhile endeavours. 


Being critical should have a more fundamental purpose and 
that is to: 


a) test and examine existing assumptions to make sure that 
they are acceptable. 


b) propose or lead the way through to solutions for any 
problems that are identified. 


In fact, the title has a ’double entendre’ contained in it. 
My pocket Oxford Dictionary defines ‘‘critical’’ as: 
“of or skilled in or given to or providing criticism, 


of the nature of a crisis, decisive for good or ill, marking 
transition from one state to another, (of nuclear reactor), 
maintaining a self-sustaining chain reaction.”’ 


| think the intention of this conference was to be skilled in 
providing criticism. However, while | think crisis would be too 
strong a word to describe the current situation, a marked 
transition (or parting of the ways) is almost upon the marketing 
research profession. Whether this will become a self-sustaining 
chain reaction or even a nuclear explosion remains to be 
seen! 


It's easy to be critical, but very difficult to create and 
implement, positive programmes. However, as a profession, | 
think we need to be more positive, for only in this way will we 
generate our own internal self-confidence and gain respect from 
others. 


| am going to sound like Jean Chretien, when | say that: 


1. We should stop bitching and work harder to improve our 
methods. We should participate more in the real world, 
rather than standing aloof from it, for only in this way will we 
raise our status in the general community. 


2. We should not be afraid to praise our competitors, rather 
than always trying to undermine or discredit their efforts. 


Ultimately, successful competitors will enhance the image 
of the whole profession (ourselves included). Knocking 
other companies may reap short term rewards, but has 
damaging long term consequences. 


competition is important and can be very productive, if 
pursued in a healthy vein: 

* it sharpens our senses 
it keeps us on our toes. 


* 


The Transition 


| would like to return to the definition of the word ’critical’, from 
the point of view that it 'marks the transition from one state to 
another.’ 


Whether research as a profession will progress from one state 
to another, | know not. However, research is certainly at a cross 
roads. We are now at a point, where two aspects of research are 
likely to diverge. | refer to: 


1. The planning and execution of research. 
2. The execution of such studies (the ’bit’ in the middle). 


Cedric Jennings in his paper on ‘‘Purchasing the Research’’ 
has some interesting proposals, which relate to this very 
distinction. 


This kind of division has been with us for some time, but is 
likely to become more and more important over the next 
decade. 


The main reason will be the onward march of technology. The 
capabilities are already with us, but it will take some time before 
they are widely utilized. 


The advent of the micro-computer and the interactive capabili- 
ties that it affords, will ensure that data collection and computer 
analysis will become more mechanical and streamlined (in terms 
of turn around time). It is also possible that such data may be 
gathered a lower cost, although this is in doubt at this point in 
time, given the scale of capital investment required. 


Thus, the execution of studies will increasingly become the 
sole responsibility of the larger research ’factories’ - the number 
crunchers! 


On the other hand, the very complexity of the communication 
process, of the data gathering revolution, the overwhelming 
amount of data available overnight to any one who wants it, will 
open up opportunities for the skilled interpreter or consultant. 


We've already seen the success of research companies in 
Canada which emphasise their executive or consultancy skill, 
rather than concentrating only on the execution of studies. 


These companies appear to have prospered in the recession, 
as buyers look for extra value. 


Two Very Important Issues 
The preceeding discussion raises two very important issues, 
which | believe will be touched on in part by Chris Commins in 
his paper on ‘‘Assessing the Quality of the Research Project’. 
These two issues are: 


1. cost vs value 


2. Information gathering vs decision making (i.e. the ‘role 
of the researcher’). 


These issues are to a large extent interrelated. 


In my view, ‘low price’ does not necessarily mean ‘good 
value’. The cheapest research is often the most expensive in the 
long run. We should all beware of the research buyer, who 
continually buys using price as the sole criterion. 


This leads into another issue, which has become more and 
more contentious with the leftward drift of most modern industrial 
societies, that of profitability. 


Profit is not a dirty word, at least not in my vocabulary. It is 
an essential ingredient for a successful, healthy industry, which 
is what we all need, buyers, and suppliers, alike. 


A profitable research industry has a number of inherently 
appealing attributes, which are to our long term good: 


1. It will attract talented people to participate in and develop 
the industry. 


2. ‘It will carry out the requisite and proper quality control 
procedures. 


3. It will re-invest in technique development and seek to 
upgrade the quality of existing procedures. 


Unprofitable companies will ultimately go out of business, but 
in the meantime they will cut corners and get us all a bad 
name. 


So beware the seductive tones of the buyer who says ’Can 
you give me a cheap quote?’ 


It’s tempting in an environment such as this, where survival 
is often more important than profitability, but it’s not a medium 
or long term route to success. 


Buyers should understand this, so should governments. 


This leads on to another vital issue, that of co-operation 
between buyers and suppliers. 


* I'd like to see buyers and suppliers working together more, 
helping each other to get on, to be successful and, of course, 
to promote research to all the important target groups. 


The success of one ’side’ is unlikely to happen in isolation 
from the success of the other ‘side’. 


Indeed, this concept of sides is divisive. We all work in the 
same industry and many people work as both buyers and 
suppliers during the course of their careers. 


* A new buyers group has been formed in Toronto, which 
should be a very positive force in helpig people with common 
interests develop their talents and positions within their 
companies. 


Let us hope that the supplier bashing, cost cutting buyer (if 
that is one of Marilyn Sandler’s stereotypes) will be banished 
forever! 


Buyers, who wish to progress within their companies, must 
first demonstrate the value of the research they commission, as 
well as their own personal capabilities. 


Although extra value can be obtained through the efficient 
execution of studies (in terms of timing, careful budgetting, 
etc.),the primary perception of extra value will be obtained from 
the planning and evaluation of research studies. 


How much should researchers contribute to the decision 
making aspects of marketing, how far should they go? Perhaps 
they should stand on the sidelines and be objective collectors of 
information. 


This is the subject of another debate! 


My own views are very clear on the subject. | believe that 
researchers should contribute more than they do at present to 
the marketing decision making process: 


* many researchers are very bright, intelligent, well educated 
individuals. 


* they understand the strengths and weaknesses of their 
research better than anyone. 


Care should be taken to distinguish between the “‘facts’’ and 
the ‘‘interpretation’’, but researchers should not be afraid to 
make their own contribution to the decision process. This mode 
of behaviour will have two benefits: 


1. It will improve or enhance their status in the eyes of their 
marketing peers. 


2. It willimprove their original planning and design capabilities. 
Only by participating in decisions will researchers learn 
which are the key measures and which are merely interest- 
ing academic exercises. 


Perhaps central to this debate, on the role of the researcher 
and his or her participation in the marketing decision process, is 
the parting of the ways mentioned earlier. 


The hewers of wood and drawers of water - the data gather- 
ers, can remain objective purists. The opportunity is for a new 
breed of researcher - the research planner, to arise and 
demonstrate the utility of all this information. 


Executional Aspects 


However, we must not forget the invisible side of research, 
the executional details. 


It is not just a question of keeping the client happy. Two tickets 
to a football game may keep a client happy, that may indeed be 
good business, but it’s not marketing research! 


We must have confidence in the invisible executional aspects 
of our studies, even though the eventual client is unlikely to see 
or even notice the results of some costly improvements to the 
basic procedures. 


Ultimately clients will recognise that research is not leading 
through to the right decisions or to the optimum directions, even 
if they cannot see the methodological problems in any particular 
study. Marketing research in general will be discredited. 


Many decisions on or choices between products or advertising 
Campaigns are based on differences of 10% or less. With 
relatively small sample sizes, it would be quite possible for a 
small number of badly conducted interviews to lead through to 
the wrong or a less than optimum choice. This small margin can 
be extremely important, it can be the deciding factor. 


AO. 


It's not enough to assume that errors will always cancel each 
other out at the extremes. They may well do so with sample sizes 
of 1,000 or 2,000, but what about the 100 - 300 sample sizes 
that we commonly work with? 


It’s our responsibility to control and to continually upgrade our 
executional procedures. Indeed, it’s in our self interest! 


This leads us on to the whole area of fieldwork, of collecting 
the data, which is the whole basis of what we do. 


The current system concerns me greatly. It is reminiscent of 
the Feudal Baronies that existed in England in the Dark Ages. 
The regional or local supervisors are often independent sup- 
pliers, answerable to no one, able to pick and choose the work 
they want to do (at least they are able to choose until the 
recession). They are often beyond control and head office quality 
control measures cannot make up for the basic lack of training 
of interviewers in the field. 


It's expedient to ignore the problems, unprofitable to pursue 
them. 


* the client will never know the difference and 


* anyway, 1,000 interviews with a few rough edges is prefera- 
ble to 500 superbly conducted ones. 


In point of fact, the comparison is probably more one of 1,000 
interviews, of which only 900 are reasonable (the other 100 
could change the direction of a marginal result and lead to a 
‘wrong’ decision), with 800 superbly conducted ones. | know 
which I'd choose. 


The whole structure of the field work industry needs reviewing 


from top to bottom, but who would be brave or foolish enough 
to attempt such a task? 


The Role of the PMRS 
Well, that’s the role of the PMRS: 


a) To pursue market development, rather than maximise 
brand share. 


b) To raise and tackle issues, which individual companies 
could not profitably pursue. 


c) To continually raise standards, where usual commercial 
accounting would lead to a continuation of the status 
quo. 


Ultimately, our own status will improve and so will the 
profitability of the companies in the industry. 


| think that we should be continually questioning and attempt- 
ing to upgrade our methods. They'll never be perfect, but that’s 
not an excuse for ignoring the problems. 


Of course, it’s a trade off, between: 


* Cost and quality 
* The practical vs the ideal. 


In addition, | know we'll always fall short of our ideals, but, if 
we aim high enough, the fall short position will be quite accepta- 
ble. If we aim too low in the first place, our fall short position will 
not be acceptable, it may even be disastrous. 


Ultimately high quality methods will prove cheaper, in the long 
run once the procedures become habitual. 


Basically, it’s all a question of education and of investment. 


* We must all get together to educate the buyers of 
research. 


* They must educate the suppliers of research. 


* Jointly, the buyers and suppliers of research must educate 
the marketing people - to accept higher costs if need be, but 
certainly to demand quality and involvement from their 
researchers. 


Behind all this, of course the interviewers and field supervisors 
must be properly and continually trained and educated. 


In additon to the educational processes mentioned above, 
there are at least two opportunities for the PMRS to set up formal 
procedures: 


1. The first is to attract and train the very best university 
graduates. We must attract talented people into our profes- 
sion. We must make it easier for new graduates to acquire 
the 2-3 years experience, which makes them such valuable 
commodities. 


2. The second is to revitalise the experienced executive - the 
person who with 10-20 years experience thinks he or she 
knows it all and is resting on his or her respective laurels. 


There are new ideas out there, which can be helpful and we 
should all be exposed to them. 


With this aim in mind, we should forge greater links or at 
least start a dialogue with academia. 


Finally, the NEP (National Energy Programme) for marketing 
research - we must not always rely on other countries (the USA 
or the UK) for technical or technique innovations or the develop- 
ment of human talent. 


It can and should be done right here in Canada. 


af 


AN OFFICIAL WELCOME TO THE SYMPOSIUM 


Martin B. Wilk, 

Chief Statistician of Canada, 
Statistics Canada, 

Ottawa, Ontario. 


Good afternoon, Ladies and Gentlemen, and welcome to this 
1982 Symposium on Survey Research. 


The theme of this Symposium is a ‘‘Critical look at survey 
research from planning to evaluation’’. A critical look at our 
collective activities in this area is, currently, not only timely but 
essential. All sectors of Canada are presently feeling the effects 
of many economic and social problems: notably continuing high 
rates of inflation and of unemployment, with declining economic 
output, underutilized productive capacity, and high interest rates, 
with attendant human and social difficulties. 


The need for knowledge and understanding of the forces and 
stresses and mechanisms at work in our society has never been 
greater. Paradoxically, most of us are, at the same time, 
constrained in our information gathering activities by the need to 
restrain expenditures. 


Clearly, the circumstances challenge us to maximize the 
benefits of every activity we undertake. At Statistics Canada we 
are increasing our efforts to exploit fully the potential of existing 
bodies of data. We are investigating to what extent our informa- 
tion gaps can be bridged without additional survey vehicles, by 
analytical and modelling processes, by statistical inference, by 
plausible imputations or by creative statistical assumptions. We 
are asking ourselves whether we are doing as much analysis of 
relationships in existing data as we should be doing. In short, we 
are trying to stretch the dollar by stretching the data. 


Concomitantly, Statistics Canada is oriented and committed to 
consulting, with users and potential users of our Statistical Data, 
in doing our survey planning, in order to establish, and to do our 
best to satisfy, their priority requirements. We are committed 
also to disseminating the results of our statistical and analytical 
work as widely as possible to serve public interest requirements 
of Canada’s governments, institutions, businesses and people. 


Despite the general recognition of the complexity of society’s 
problems and opportunities, and the evident need for better 
understanding as a basis for effective planning and actions, it is 
unlikely in the current fiscal/economic circumstances that 
resources for informational development will increasesd in 
correspondence with that need. It is therefore all the more 
necessary that survey researchers and statisticians achieve 
maximum exploitation of both new and existing sources and 
vehicles of information 


That is what this symposium is about. It is an opportunity for 
government and commercial survey researchers to confer in a 
professional rather than a business setting. We have different 
responsibilities, and perhaps different orientations. At Statistics 
Canada we are producing statistical data to meet a board 
spectrum of purposes, from Government policy-making to gene- 
ral public education requirements. 


Those of you who are in the private sector are geared more 
toward market analysis and attitudinal surveys. But we can 
borrow from each other, share our tools with each other. We 
have a commonality of expertise and of problems, an equal 
interest in the techniques, principles and practical approaches of 
survey research. The next few days will be devoted to discus- 
sions of survey strategies and practices and | have every 
expectation that this Symposium will contribute to achieving 
increased effectiveness in the processes of statistical informa- 
tion development. 


| would like to thank the organizers of this Symposium for the 
excellent program they have put together. Specifically, | want to 
express our thanks to Ted Keller and Alastair Lynn of the 
Professional Marketing Research Society, and to Merv McCa- 
mon and Ed Wilhelm of Statistics Canada. 


It is my pleasure to have this opportunity to open the sympo- 
sium and to welcome you. | believe you will benefit from the 
presentations and discussions planned for this symposium, and 
| trust you will also enjoy the informal interactions and your visit 
to Ottawa. | wish you a productive and stimulating session. 


Thank You. 
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THE UTILISATION OF KNOWLEDGE IN THE DECI- 
SION-MAKING PROCESS 


Vincent P. Barabba 
Director, Market Intelligence, 
Eastman Kodak Company 


Abstract 


This paper outlines the process which is being adopted at 
Eastman Kodak to make marketing decisions, using the 
“knowledge utilization’’ concept. Knowledge utilization is de- 
fined as the conversion of raw data into an ultimate action. A 
small number of teams or study groups are assembled, each 
representing a different point of view on a particular marketing 
situation. These teams use available market data to support their 
respective adversarial positions. The system of reconciling these 
conflicting viewpoints to form a consensus is called ‘‘Stategic 
Assumption Surfacing and Testing’, or ‘‘S.A.S.T.”’ 
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Thank you ... I’m delighted to be with you. I’ve already seen 
many old friends and familiar faces here in Ottawa ... the 
presentation I’m bringing you today is really an update -- | think 
you'll find it’s an exciting update -- on some basic concepts that 
| initially discussed here in 1978 ... at another major conference 
jointly sponsored by Statistics Canada. 


At that time, | talked about the concept of knowledge utilisation 
-- and how we were trying to implement it within large public and 
private sector organizations. 


Today, I'd like to bring you up to date on what’s been 
happening since then. It reminds me of the story of two major 
league baseball players -- a pitcher and a shortstop. 


One day, they got to talking about what happens to people 
when they die ... which led them to the question: ‘‘Is there 
baseball in heaven?” 


Being good friends, they decided then-and-there to make a 
pact: Whoever died first would come back to earth and tell the 
other one whether or not there was baseball in heaven. Not long 
after, the shortstop died, went to heaven, and -- true to his word 
-- came back to earth as a ghost to answer this crucial 
question. 


“Well,’’ he said, appearing to the pitcher in the locker room 
after a game, ‘| have some good news and some bad news. The 
good news is that there is baseball in heaven.” 


“What's the bad news?”’ asked the pitcher. 


“The bad news,”’ said the ghost, ‘“‘is that you’re pitching on 
Saturday.”’ 


So much for effective knowledge utilization in baseball! 
On a more serious note, it does seem as if knowledge 


utilization is something everyone talks about -- but few know how 
to implement it in the complex environment of the 1980's. 


Knowledge utilization -- in its simplest definition -- is the 
conversion of raw data into an ultimate action. | like professor 
Jerry Zaltman’s description of the critical factors involved in that 
conversion process: 


“Knowledge utilization is the broad process by which the needs 
of research users are determined and communicated to resear- 
chers. This, in turn, leads to research designed to meet these 
needs ... and eventually, to new knowledge based on research. 
This is communicated to users, who apply this knowledge to 
answer their needs.”’ 


As part of my 1978 presentation, | discussed a conceptual 
framework that addressed the ways that knowledge utilization 
could be implemented more effectively in the decision-making 
process within large organizations. 


In today’s presentation, I'd like to focus on the ways we’re 
actually implementing this conceptual framework within a major 
corporate setting. 


Here, of course, I'll be drawing primarily on my own experien- 
ces as director of Market Intelligence for the Eastman Kodak 
Company. 


First, let me put that word ‘“‘intelligence’’ into a context that | 
find useful, and distinguish it from other connotations that range 
from the covert operations of the CIA ... to the high 1.Q.’s of 
MENSA members. For our purposes, it’s useful to think of the 
word ‘‘intelligence’’ in terms of three separate meanings, de- 
pending on the context: 


- First, we can use the word ‘‘intelligence’’ to mean a type of 
organization: 


Market intelligence is chartered to gather, analyze, and 
provide timely knowledge to operating groups on the basis of 
their information needs. 


- Second, we can consider ‘“‘intelligence’’ as a type of 
activity: 


The intelligence process to support the market plan is both 
comprehensive and timely 


- Third, we can consider ‘“‘intelligence’’ as a type of 
knowledge: 


For example, this intelligence describes how key elements in 
the distribution channels will react to a new product 
concept. 


In today’s presentation, you'll see how we apply the word 
‘intelligence’ in all three ways within Kodak in our efforts to 
achieve effective knowledge utilization through our Market 
Intelligence System. 


Our efforts, in fact, are a reflection of an important linkage 
within the company ... between the Market Intelligence function 
and top management's recognition that today, more than ever, 
effective knowledge utilization (or relevant information) is a key 
factor in meeting its corporate objective. 


That objective can be summarized this way: ‘‘To sail ahead of 
the winds of change’”’ in an increasingly competitive environment 
on a worldwide basis. 
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Relevant information, as defined by Kodak Management, is 
accurate, timely, and easy-to-use. Its value comes when it can 
have an impact on business decisons as they arise. 


In support of this concept, Kodak management has organized 
the role of Market Intelligence on a worldwide basis ... in addition, 
a dedicated computer system is designed to create a powerful 
data-base management tool to support all the functions of 
Market Intelligence. 


Further, this computer system is designed not only to track 
short-term conditons, but also to assist us to ‘‘look forward’ as 
required by the long-term planning process. 


Now, let me share with you a fuller treatment of the Kodak 
Market Intelligence System ... or, as it’s called: The MAIN 
System. I've found that the easiest way to define the Kodak MAIN 
System is with an illustrative model of how it is structured. 


Kodak 
MAIN 
System 


Organization’s 


Marketplace 


Decision Reality 


Process 


First, let me describe the basic framework within which the 
MAIN System functions. At one end of the horizontal plane, we 
have the organizations’s decision process. This reflects what the 
organization is capable and willing to present to the marketplace. 
At the other end, we have marketplace reality. 


This reflects what the market needs or says that it needs. It’s 
important to point out that this horizontal dimension can be 
applied not only to to the private sector -- to organizations like 
Kodak ... but also to the public sector -- to government program 
planners and their constitutencies. And I’m sure you can think of 
some examples. 


Kodak 
MAIN 
System 


Accountable 


Management 


Organization's 


Marketplace 


Decision 


Reality 


Process 


Market 


Intelligence 


Now, I'd like to look at the vertical dimension of the MAIN 
System: At one end is Accountable Management. These are the 
people responsible for the decisions about what the organization 
will present to the marketplace ... 


Thus, accountable management are the people who deter- 
mine what information is needed. 


In other words, doing the right things. 


At the other end is Market Intelligence. These are the people 
responsible for the reliability and validity of the information about 
the marketplace. Thus, it’s market intelligence that determines 
how the information will be acquired and presented. 


- In other words, doing things right. 


Now, let’s place four of the five primary market intelligence 
functions within this framework. 


Kodak 
MAIN 
System 


Accountable 


Management 


Analyse/ 
Describe 
Market 

Information 


Assess 


Market 


Information 


Needs 
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Marketplace 
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Store, 
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Marketplace 


Display 
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Intelligence 


Function 1 is to Assess Market Information Needs. This 
involves determining market information requirements within a 
worldwide scope. In assessing market information needs, we 
must focus on priority assumptions that are uncertain, as well as 
monitor key factors that are critical to the success of selected 
marketing plans ... Obviously, is a very important function, and 
I’ll return to describe it more fully in a few moments. 


Function 2 of the MAIN System is to Measure the Marketplace. 
In other words, to gather relevant and valid data. 


Function 3 is to Store, Retrieve, and Display Market Data. This 
utilizes the powerful computer data-base management system 
we mentioned earlier. 


Function 4 of the MAIN System is to Analyze and Describe 
Market Information. 


There’s an essential distinction to be made here: Within 
Function 4, it’s role of Market Intelligence to use analytic 
techniques in order to describe market information for use by the 
planning community ... but it is management planners who must 
develop the corporate policy analyses. These two form the basis 
for the ultimate business decisions of Kodak. 
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Now, we add a fifth function to the MAIN System. This reflects 
all the ways we evaluate the impact that the information has had 
on Kodak’s decision-making process ... that information, as 
we’ve noted, was identified, collected, and analyzed during the 
first four functions. 


Kodak 
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System 
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Market 


Information 


Market 


Information 


Needs 
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Impact Marketplace 
Decision 
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Retrieve, 


Display Marketplace 
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Intelligence 


This last function is represented by the diamond. It’s positio- 
ned central to the overall structure and the initial four functions. 
It reflects the broad-based participation and interaction required 
to make the whole process work. 


In summary, then, you can see that the Kodak MAIN System 
is designed to: 


1. Develop a priority list of market information needs. 
2. Gather required worldwide market information. 


3. Store, retrieve, and display these data in ways that facilitate 
analysis and linking with other information. 


4. Analyze and describe market information: prepare descripti- 
ve analyses for use by decision makers in developing policy 
analyses. 


5. Evaluate the impact of the MAIN System on the decision- 
making process. 


At this point, | imagine there’s another key question you may 
be asking: 


What are the criteria for success in evaluating the Market 
Intelligence function within Kodak? 


In my view, management decision-makers will evaluate the 
success of Market Intelligence by its capability to respond to their 
needs. 


That means Market Intelligence must provide relevant infor- 
mation at any point between the following two positions: 


1. ‘The decision | face requires the accuracy and precision of 
a new, specific-purpose study. | know this study will be 
costly and the results won't be available immediately, but 
the importance of the decision to the Company warrants the 
delay and expense...’’ 


2.  ‘‘Because of the type of information | need and the immedi- 
acy with which | need it, I’m willing to trade off accuracy and 
precision for speed and, possibly, a reduced cost. There- 
fore, I'd be willing to further analyze existing information 
and use this is a surrogate for the specific information | 
need.” 


As | mentioned earlier, a major objective for the MAIN System 
is to provide forward-looking information ... that is, a system that 
allows Kodak to anticipate and plan for changes in the market- 
place -- rather than finding out about them after the fact, and 
being forced to react. 


Now, I'd like to show you a series of three diagrams to 
illustrate how the sum of these five functions -- the MAIN System 
-- works in the day-to-day encounters of Kodak and the environ- 
ment in which we operate. 


That process typically begins when the ‘‘issue”’ is identified by 
events occurring in the environment. The ‘‘messages’’ about 
these factors came directly to us from the marketplace through 
a variety of communications channels. Often, we had not fully 
anticipated them. 


Formerly, when there was time ... and if it was very important 
. a specific-purpose survey was conducted to bring more 
understanding to our decision-process before we responded. 


The MAIN System, represented here by the box, is now in 
place and will have a significant impact on this process ... and the 
extent of that impact will occur over time. 


Here we can see the beginning of change: As is the case 
today, most of the issues will still come directly to Kodak from the 
marketplace ... in certain instances, we’ll conduct a specific- 
purpose study -- but, at the same time we're building our 
information base. That means, before any study is carried out, 
the objective of the research will be established and reviewed. 
This will be done in the context of what other (general-purpose) 
needs could also be addressed by the study. 


The results will be stored according to the issues they relate 
to -- rather than the traditional project classification. Simulta- 
neously, we will be developing general-purpose data bases and 
analytical tools for further analysis. 


We will also measure the marketplace reaction to the Kodak 
response: That is, what actually happened, versus our 
estimates. 


The significant difference is the two-way flow of information 
about the environment -- coming from, as well as to, the MAIN 
System. 


That shift signals our response to management’s forward- 
looking statement of information needs in terms of effective 
knowledge utilization. 


| must also point out that this does not mean that all informa- 
tion and signals will come from the MAIN System ... it does 
mean, however, that we will be better able to anticipate change 
and, therefore, to manage our response more effectively. 


Obviously, corporate commitment is needed to achieve this 
objective: that is, a commitment to a planning framework that 
clearly states company objectives and their underlying 
assumptions. 
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In the remaining time | have with you today, I'd like to focus 
your attention on this corporate planning framework ... specifi- 
cally, on one facet -- the importance of understanding the 
underlying assumptions that can influence management action. 


I'd like to share with you a process that is giving Kodak added 
insights about the impact of these assumptions on knowledge 
utilization in the decision-making process. 


I’ve chosen to focus on this area for another reason as well. 
It seems obvious to me, if we study the agenda for this confer- 
ence, and relate it to the five functions of the MAIN System ... 
that we're assured of gaining many valuable insights related to 
functions 2, 3, 4 and 5. 


For that reason, | would like to concentrate on what | consider 
the most difficult and challenging of these ... namely, function 1: 
the assessment of market information needs. 


You'll recall my earlier promise to return to this important area 
... SO, let’s begin by looking at what | see as the major challenge 
for knowledge utilization during the 1980's: 


To avoid coming up with the right solutions to the wrong 
problems. The eminent statistician John Tukey once put it this 
way: 


“Far better an approximate solution to the right problem ... 
than an exact solution to the wrong problem.” 


We’re also familiar with the old saying,‘‘a problem well- 
defined, is half solved.”’ 


Or, as I’ve expressed it from time to time: ‘‘A problem 
ill-defined will never be solved.” 


Let me underscore the fact that many of you know only to well: 
In today’s competitive marketplace, we’re dealing with highly 
complex problems ... in an environment where there is a 
diminishing margin for error. 


These complex problems can often be recognized by their 
shared characteristics. They tend to be highly ambiguous ... they 
lack definitive formulation ... they have many levels ... they’re 
essentially unique, with no clear beginning or end ... and they’re 
marked by a large degree of uncertainty. 


Because there is so much uncertainty, there are, naturally, 
many assumptions that have to be made in developing solutions 
for these complex problems. Part of the solution is to be able to 
identify key areas where information is needed in defining the 
ultimate business strategy. 


At the same time, millions of dollars can be riding on these 
strategies. It’s critically important that the underlying assump- 
tions are clearly defined ... as part of identifying market informa- 
tion needs. 


The concept we are using at Kodak has been designed to do 
just that. 


It's called S.A.S.T. ... which stands for Strategic Assumption 
Surfacing and Testing. The thrust of S.A.S.T. is to surface and 
challenge the underlying key assumptions, and the related 
market information needs, on which every business plan must 
rest. 


What I'd like to do now is take you through a very brief version 
of the S.A.S.T. process, using a composite Kodak case history 
to show you how S.A.S.T. can be applied in today’s 
organizations. 


It’s become increasingly clear to me, over the years, as a 
consultant and a manager that the real difficulty top management 
faces when it comes to planning ... is to explicitly identify the 
assumptions they hold ... and to sort out the faulty ones from the 
sound ones. 


As you know, it is not unusual for managers to spend little or 
no time examining the particular set of assumptions they hold. 


A major reason stems from the fact that much of the planning 
activity in North American business today is aimed at coming up 
with the right solution ... not to (quote) ‘‘waste time’’ on 
examining whether or not we’re asking the right question. 


The S.A.S.T. concept -- through a carefully planned sequence 
of steps -- can help planners define the right problem ... by 
surfacing and testing the underlying assumptions. 


It also helps identify areas that need additional information 
activities to reduce uncertainty ... and, it’s a concept that 
works. 


It played a major role in the US Census Bureau decision not 
to adjust the 1980 census count and, it’s been used successfully 
within Kodak in conjunction with recent projects that include ... 
needs assessment for a whole line of business ... for a major 
camera program ... and a new development in the industrial 
area. 


We also used the process to develop the plan for implemen- 
ting the Kodak MAIN System itself! In addition, the process is 
being designed into future planning efforts for the company ... | 
might also mention that there is a small but growing body of 
literature dealing with this concept ... two of my colleagues -- 
Richard Mason now at the University of Arizona and lan Mitroff 
at the University of Southern California -- have collaborated on 
recent publications.” 


| should emphasize one other point about the S.A.S.T. 
concept. 


It is an adjunct and not a replacement for the exercise of 
creative managerial judgment. In fact, it complements such 
judgment. 


Now, let’s look at the five key steps in the S.A.S.T. process. 
They are: 


1. Formation of separate groups to develop different assump- 
tions, or perspectives, on the problem under review. 


2. Each group then surfaces and rates its assumptions about 
the problem, using the ‘‘stakeholder method,”’ which 1'll 
discuss in a moment. 


* Mason, Richard O., and lan |. Mitroff. Challenging Strategic 
Planning Assumptions: Theory, Cases and Techniques. 
School of Business Administration, University of Southern 
California, Los Angeles, California. 
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3. The members of each group then debate the assumptions 
among themselves and rank them according to two criteria 
... the relative importance of each assumption ... and the 
relative certainty of each assumption. 


4. The debate continues between groups ... and the partici- 
pants work to reach agreement on what areas need addi- 
tional information ... especially when an important assump- 
tion is seen as not having much certainty. 


5. Finally, all of the participants engage in a collaborative effort 
to synthesize the results and arrive at a consensus. 


Now, let’s review each step more closely ... and then we'll see 
how each one applies to a composite case history from Kodak. 
This particular example has been altered to fit this afternoon’s 
presentation needs, but it’s based on an actual corporate 
decision that faced Kodak management. 


The first phase -- group formation -- is an especially critical 
element. It encompasses three basic objectives: 


Each group must be as internally homogeneous as possible 

. each group must be as different from all other groups as 

possible ... and, all the groups -- collectively -- must represent 

broad coverage of the different perspectives that management 
wants to bring to bear on the complex problem at hand. 


Experience has shown that the S.A.S.T. process seems to be 
most effective with a total of 18 to 24 people ... with 5 to 7 within 
each working group. 


I'd like to focus, for a moment, on the underlying reasons for 
forming groups with different orientations. First of all, putting 
like-minded individuals in the same group tends to minimize 
conflict ... while it enhances the strength of a particular outlook 
... and the strength of a particular outlook ... and the strength of 
the underlying assumptions behind that outlook. 


Secondly, this approach reverses the usual way of doing 
things. As a rule, organizations do not reward their members for 
challenging accepted ways of thinking or behaving. However, it’s 
much easier to get creative about challenging the status quo if 
one is a member of a group that thinks the same way and has 
been directed to do so. 


Incidentally, the group also offers support for challenging 
some other group. 


Group formation is essential because it means that manage- 
ment can assure that different points of view will be considered 
explicitly. 


Well, how do participants get into different groups? One 
method is to have people select themselves into their own 
groups from a list of potential strategies. Another is to form 
groups on the basis of ‘‘known”’ interests. 


Or, participants can be selected on the basis of their in- 
volvement in particular areas, such areas as research ... or 
manufacturing. 


Once each group is formed, the S.A.S.T. process requires the 
group to get started by giving themselves a name ... and after 
that, to summarize its primary point of view. 


Now, let's look at the Kodak composite case study. The 
decision management faced was real: 


Should Kodak continue 


to manufacture and sell 


amateur movie cameras? 


Again, let me condition what follows by pointing out that -- this 
example deals with a real problem -- to that extent it’s not 
hypothetical. On the other hand, this presentation has been quite 
simplified and sanitized for consumption outside the company. 


Envisage, if you will, three sample working groups: 


The first group contends that there are strong pockets of 
demand -- more so in Europe, and particularly in France, for 
amateur movie cameras. They call themselves ‘‘Viva La 
France’’. Here’s where our international marketing people 
naturally aligned themselves. 


The second group asks -- ‘‘Does It Sell in Peoria?’ They've 
adopted the position that the US market is slipping and Europe 
will soon follow. You'll find the US marketing people drawn here. 
They say: let other manufacturers supply cameras, while Kodak 
continues to sell movie film and processing to new and current 
users. 


The third group -- calling itself ‘‘Half A Loaf Is Sufficient’’ -- has 
attracted the participants with manufacturing interests. They 
contend that the US market is not dead ... only at a low 
replacement position. They also say that Europe is strong ... and 
the rest of the world is coming to life. Their premise is ... we can 
continue to manufacture movie cameras ... make only half as 
much profit -- and still be doing okay. 


They also say: Hang in there ... and reap the re-birth with 
products designed for the developing countries. 


Now, let’s return to Phase Il of the S.A.S.T. process ... 
assumption surfacing and rating. In this phase, the groups meet 
separately to generate their assumptions. They do this by 
utilizing the stakeholder method: In brief, stakeholders are all 
those claimants -- both inside and outside the firm -- that have 
a vested interest in the complex problem and its solution. This 
list is suggestive of the types of stakeholders that could be 
involved: 


¢ Creditors 

e Energy sources 

e Labor Unions 

e Federal gov't agencies 
e Research 

e Stakeholders 
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Another factor that’s worth remembering is that this part of the 
S.A.S.T. process represents a different way of looking at things 


The task requires people to go from strategy back to assump- 
tions ... rather than from assumptions to strategy. 


In addition to identifying the obvious stakeholders, each group 
must also find one or two ‘‘low probability/high risk’ stakehol- 
ders. The reason? This kind of stakeholder is easily overlooked 
-- yet it can have a significant impact on carrying out a plan. 


In generating their assumptions, each group must ask the 
question: ‘‘What must we assume about this stakeholder and its 
future behavior in order for the plan to be successful?”’ 


Then, as a way to test the importance of each assumption, the 
group is told to generate a counter-assumption. That opposite 
view is what planners call the ‘‘deadly enemy’’ to the stated 
assumption. If the ‘‘deadly enemy’”’ does not appear to have any 
significant impact on the strategy, then it’s likely that the original 
assumption is not very relevant to the problem. 


Next, each group narrows its large lists of assumptions and 
counter-assumptions down to 10-15 ... and ranks them in 
importance. 


These key assumptions are then plotted on an importance- 
plausibility graph. And the eventual premises of the plan, of 
course, are those assumptions that are presumed to be most 
important and most plausible. 


Briefly, let me point out that plausability refers to the credibility 
-- or reasonableness -- of an argument. It’s quite different from 
probability -- which refers to the likelihood that certain events will 
occur. 


The concept of plausibility is important, because in many 
situations the decision-maker may have to take action before 
knowing the probability with which the events contained in the 
argument may happen. This means rating the plausibility of the 
argument. 


Finally, each group settles on its pivotal assumptions and lists 
them in order of priority. 


These are the basic assumptions, in the group’s judgment, 
that should become the fundamental premises of the policy. 
Each group identifies six to eight pivotal assumptions ... which 
is a critical part of the strategy-making process. 


In phase Ill of S.A.S.T., the groups come together to debate 
and further test their assumptions. 


Each group makes a brief presentation of its position. No 
debate is permitted at this step. Then, the individual groups meet 
alone, once again. Their task is to identify the assumptions from 
the other presentations that do the most damage to their own. 


Each small group reports these damaging assumptions to the 
group as a whole. From this input, the participants make a list of 
all the damaging assumptions from all the groups. In effect, this 
is a final set of critical assumptions. 


The participants list these assumptions in the order of priority 
... and the debate begins and continues until all groups feel they 


have identified all of the critical issues resulting out of the 
debate. 


And while they may not have resolved the differences between 
them, they certainly have a better idea of why they are 
divided. 


It’s time, once again, to look at our Kodak case study. Let’s 
see how our three groups interacted so far. Our Viva La France 
group had assumed that: In Europe, consumers still want to take 
home movies. In support of that premise they said ... sales of film 
are stable indicating people still use their movie cameras. 


The Peoria group rebutted that one by saying: ‘‘The market for 
cameras will slip ... because the European birth rate is declining 
-- and everybody knows you take home movies of babies!’’ 


Since this assumption was not damaging to their strategy, the 
Half A Loaf group chose not to make a rebuttal. 


Our second group -- Does It Sell in Peoria -- listed the 
assumption that -- Consumers in the US no longer want to take 
home movies. 


Their support: 


1. Sales of movie cameras are declining. 
2. Consumer attitudes are ‘‘anti’’ home movie. 


3. The US birth rate has dropped -- removing the subject of 
most home movies. 


4. Other forms of video capture and display are preferred. 


The Half A Loaf contingent rebutted points ‘‘2’’ and ‘'4’’ by 
saying: 


1) Not all consumers have anti home movie attitudes -- we still 
are selling a lot of film and some cameras. 


2) We can design new camera which can be sold even less 
expensively than other forms of video and with some 
improved features. 


Viva La France also rebutted by saying: 


European and Japanese manufacturers are still successful in 
making and selling cameras in the US and Europe, while 
establishing new markets in other parts of the world (some with 
rapidly increasing birth rates) to insure further growth. 


Our Half A Loaf group said: People will buy a low-cost movie 
camera -- and we can make one. They supported that in three 
ways: 


The technology is available ... many people think the price of 
today’s home movie cameras is too high ... and young people 
are more visually oriented than ever. 


To rebut that premise, the Peoria group said: Long-term sales 
will not be sufficient to support capital cost for a new camera line 
... and make profits, too! Besides, other manufacturers are filling 
consumer camera needs ... so there will be continued demand 
for existing film manufacturing capacity. We can use that additio- 
nal capital investment for higher return activities.’’ 


The next step in the process is to list the issues that arise from 
the debate ... and to attempt to achieve group consensus on 
actions required for each one. 
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Here’s where phase four of S.A.S.T. comes into play. That’s 
the analysis of the information requirements to resolve the highly 
important assumptions that carry low levels of plausibility. 


In general, these information-producing activites fall under 
three major approaches that you’re all familiar with: 


First is a basic research approach. 
Second is a judgmental approach. 
And the third is monitoring. 


The cost of obtaining such information can be quite high. But 
S.A.S.T. suggests a useful approach -- called ‘‘a Manager’s 
Planning Book’ -- that can help policymakers deal with this 
problem. Among other things, at the close of the process, the 
book will contain: 


1. A prioritized list of the most critical issues management 
faces ... as revealed by S.A.S.T. 


2. An assessment of the current state of Knowledge with 
respect to those issues. 


3. A list of current and planned information-producing activi- 
ties that are designed to improve the state of knowledge 
relevant to the critical issues. 


Returning to our Kodak example, for a moment ... we can see 
that our groups reached a concensus that the information needs 
should be targeted around ‘‘consumer attitudes on home mov- 
ies.’’ They determined the most effective techniques for obtain- 
ing this data included the judgmental approach via focus groups 
... and the basic research approach, with an attitudinal study for 
segmentation within the market. 


The fifth and final phase of S.A.S.T. is synthesis. Like other 
approaches, S.A.S.T. does not guarantee that synthesis will 
occur. But, if nothing else, the steps in S.A.S.T. improve the odds 
that synthesis can be reached. 


Let me show you why. Up to this point, the process has 
involved very carefully controlled conflict. Now, in contrast, a 
mood of collaboration is sought. 


In this way, participants use the process to attempt consensus 
on an unresolved issue. 


It's worthwhile to briefly examine the ways that conflict can be 
resolved, since S.A.S.T. attempts an approach that differs from 
‘traditional’ means. There are at least five general ways to 
respond to conflict: 


- Avoiding. Both parties choose to avoid the conflict 
entirely. 

- Competing. One party wins everything. 

- Accommodating. One party gives in entirely. 


- Compromising. Each party receives half of what they 
want. 


- Collaborating. Both parties contribute. 
Let’s look at this graphic treatment of the ‘‘traditional’’ form of 
resolving conflict. 


This line, between ‘‘competing’’ and ‘‘accommodating,”’ is 
punctuated in the middle by ‘‘compromising.”’ It is structured so 
that one of the parties must ‘‘win’’ or ‘‘lose’’ a position. 


This is called a ‘‘distributive’’ dimension. 


But the “‘integrative’’ dimension moves upward on the 
“avoid’’ and ‘‘collaborate’’ line ... 


The objective of collaboration is directed toward an integration 
of assumptions ... calls for synergistic solutions ... and results in 
both parties ‘‘winning’”’ -- a win/win objective. And, it’s based on 
the recognition that the sum of the resources available is not 
necessarily fixed ... in contrast to the ‘‘traditional’’ win-lose 
approach. 


Let’s take a moment and look more closely at the use of 
conflict and collaboration within S.A.S.T. ... at the outset, you'll 
recall, the process deliberately makes use of controlled conflict 
between the groups. 


This offers a valuable experience for the participants. That’s 
because, while most people understand that conflict exists in all 
organizations ... few know how to make creative use of these 
conflicts to produce new insights on problems. 


Later in the process, S.A.S.T. stresses collaboration, based 
on the integration of ideas. 


Again, we are only now beginning to see that people in these 
same organizations also need training for collaboration, or 
synthesis -- particularly in our culture, which does not encourage 
such thinking. 


Incidentally, an added benefit of this approach is that the 
outcome generally requires less time and money to receive final 
management approval. And, hopefully, it minimizes the loss of 
innovative ideas that could ultimately be to the organization’s 
advantage. 


The final decision, of course, is not made by the S.A.S.T. 
process. 


But management can use the S.A.S.T. results with a high 
degree of confidence because of the following conditions: 


1. The broadest knowledge within the organization was 
brought to bear on the strategic issue. 


2. A thorough, systematic, and critical examination was con- 
ducted on the various strategic options and unerlying 
assumptions. 


3. All the participants understand -- many for the first time -- 
the critical assumptions that underlie the adopted strategy 
. and, while agreement among all of the parties is not 


probable, their awareness and mutual understanding is a 
substantial element in the successful implementation of the 
strategy. 


Now we'll return to our Kodak composite case study one more 
time. 


Without going through the details of how the final synthesis 
might -- or might not -- be achieved ... I'll just tell you that Kodak 
management decided to discontinue the manufacture and mar- 
keting of amateur movie cameras. So these days, if you’re in 
Peoria ... or Paris ... you won’t be able to buy a new, Kodak 
home movie camera at your favorite store. However, if you have 
a desire for a new Kodak disc camera, | think you’ll find that one 
is available. 
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In closing, let me share with you briefly what we’re doing at 
Kodak to make this concept even more valuable in the real-life 
setting. As | mentioned, we’ve conducted many actual S.A.S.T.’s 
on current -- and complex -- problems facing Kodak decision- 
makers ... and for four of them, we have completed an evaluation 
process. The key document is an 8-page questionnaire that 
includes both open-ended questions and a semantic differential 
scale measuring 15 attributes. 


The questionnaire is distributed to each person who participa- 
tes. So far, we have analyzed 94 responses from the four 
SAS. 8S. 


We've analyzed the results both quantitatively and qualitative- 
ly ... and our analyses reveal an overwhelmingly positive reaction 
to the S.A.S.T. Process: 93 per cent of all respondents gave it an 
overall positive rating ... while only 7 per cent rated the overall 
conferences as less than favorable. 


This evaluation procedure is important for a number of 
reasons. Obviously, the results demonstrate that, from the 
participants’ viewpoint, this concept is a valuable tool for dealing 
with complex problems in a real-life setting. 


However, the purpose of our evaluation efforts was not just to 
prove whether or not Kodak people like the S.A.S.T. concept and 
think that it works. 


Our real focus is to identify the strengths and weaknesses 
within the S.A.S.T. process itself ... so that we can modify it and 
make it as effective as possible both in terms of the resources 
required ... and in terms of results that will benefit 
decison-makers. 


Now, before | tell you more about what our evaluation showed 
... 'd like to touch briefly on the methodology we used to conduct 
Our analysis. 


The quantitative part of our evaluation is comprised of a 
statistical analysis of the bipolar scale responses to 15 attributes, 
plus the overall rating of the conferences. 


These are included in the questionnaire as a semantic differ- 
ential segment. Participants are asked to rank various features 
of the conference on a scale of 1 to 9 ... and then to rank the 
overall conference, using the same scale. 


The mean overall ratings from participants in the four confer- 
ences ranged from 6.9 to 7.2 on our 1 to 9 scale ... which 
certainly shows that we’re headed in the right direction. 


In addition to this visual similarity, statistical tests also proved 
there was virtually no difference between groups or categories 
of work background, such as marketing, manufacturing, re- 
search, etc. 


The qualitative part of the analysis is a compilation of the 
responses to the open-ended questions asked in the survey. 


In analyzing these, we looked for frequently mentioned com- 
ments. We identified those that were cited by at least 10 percent 
of the respondents. We wound four strengths that stood out: 
¢ Surfacing of issues and assumptions 


¢ Exchanging information and opinions 


¢ Debating positions ... and 


¢ Building interpersonal ties. 


The two weaknesses that respondents most frequently men- 
tioned were: 


¢« The time commitment required ... and 


e The perceived lack of a satisfactory conclusion. 


Surfacing issues and assumptions is one of the primary 
objectives of S.A.S.T. ... and 60 percent of respondents cited this 
feature as one of the most beneficial of the process. As one 
participant put it ... the process made those involved “realize that 
the assumptions they had taken as ’given’ ... really aren’t.’’ 


S.A.S.T.’s ability to handle conflict and diverse positions ... 
and to provide a means for debate ... were two other strengths 
that received frequent mention. 


Here’s a typical quote: 


“The procedure gave widely-opposed views equal airing ... and 
forced those holding what will probably be the corporate view to 
think it through and vigorously defend it.”’ 


Another participant put it this way: 


“First and foremost, | was introduced to valid divergent opinions 
from my own, which is always a healthy medicine to have to 
take.”’ 


Perceptions varied, of course, and a number of respondents 
had some criticism of the debate process. Some frequently 
mentioned concerns were ... not enough facts on hand to 
support debate ... and the debate didn’t cover enough issues, 
especially those judged ‘‘neutral.’’ 


Sixty percent of the respondents mentioned team-building and 
interpersonal aspects as a strength of S.A.S.T. As one partici- 
pant wrote: 


‘“‘The process has brought a number of us closer on a profes- 
sional level, and this is always enriching.’’ 


In addition to these features, 35 percent of the respondents 
mentioned execution and organization as a strength. This 
comment may be of particular interest to any academics here 
today: 


“S.A.S.T. may have sounded a bit ’text-bookish’ -- but some- 
times textbooks are right!’’ 


One of the most valuable findings from testing S.A.S.T. in the 
real-life setting is the fact that we need to improve the synthesis 
and closure segments of the process. 


Only 15 percent of the respondents mentioned synthesis and 
closure as a strength of the process ... while 45 percent viewed 
the lack of a satisfactory closure as a weakness. Here is a quote 
that typifies many of the respondents’ concerns: 


“I’m not sure we came out with a commitment. Did the process 
in a sense artificially bring us to a consensus?” 
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| think such responses are important for two reasons. First, 
they point up the need to improve the process itself. And second, 
they show that we may need to implement a plan for longer-term 
follow-up to keep participants informed of the progress that’s 
being made on the issue. 


We've surmised that the sense of frustration expressed in 
such statements could be as much a reaction to the fact that the 
issues involved are somewhat more out of participants’ hands 
after the S.A.S.T. is over ... as they are a reaction to the process 
itself. 


If that’s the case, a report on S.A.S.T. Progress after the 


conference ... combined with changes in our approach to 
synthesis within the process -- could help alleviate this 
frustration. 


It’s also occurred to us that this perceived lack of synthesis 
could result from a lack of awareness on the part of participants 
about what the S.A.S.T. is actually supposed to achieve. 


If further research show that’s true, we are considering the 
design of an introductory presentation ... help develop realistic 
expectations. 


Another rationale for why people rated the synthesis concept 
so low may stem from the fact that S.A.S.T. is so different from 
the traditional decision-making methods they've used in the past 
... take a moment and think how difficult it must be for a person 
who is accustomed to being the decision-maker ... to suddenly 
become a peer of 23 other participants in a group process. 


This is the type of issue that isn’t readily apparent until you 
attempt to implement a ‘‘textbook”’ process such as S.A.S.T. in 
a real-life situation. 


The amount of time required for the process, as | mentioned 
earlier, was also a concern. 


Several participants made interesting points in this area ... 
here are two sample quotes: 


1. ‘A smaller group could have gotten as far as we are now 
... but the reduced time would be lost when we tried to sell 
people on implementation’’ ... 


2. ‘It may be that this process is most effective when em- 
ployee ‘buy in’ is essential and the question is not within 
management’s absolute prerogative.” 


In this very brief summary, you’ve seen an overview of the 
evaluation process we’ve used in the four S.A.S.T.’s conducted 
so far at Kodak. 


In the future, we’re planning to look at other key issues as well 
. including whether or not S.A.S.T. actually does save time 
when it comes to ‘“‘buy-in’’ and longer-term implementation. 


To sum up, then, | think you can see how we've been able to 
refine and improve the concept of S.A.S.T. through implementa- 
tion in a real-life organizational setting. 


In a similar way, we've been able to modify the original 
knowledge utilization concepts through development of the 
Kodak Market Intelligence system ... with its five functions. 


As | said earlier ... after reviewing the agenda for the rest of 
this session ... I’m confident you’re going to be gain a lot of 
valuable information about functions 2, 3, 4 and 5. 


| hope then, that, today, I’ve been able to offer you some 
insights into what is involved in Function Number One. 


Thank you. 


THE ROLE OF MARKETING RESEARCH IN PLAN- 
NING AND EVALUATION 


Mr. G. Chiasson, Vice-President, 
Passenger Planning and Marketing, 
Air Canada, Montréal, Québec. 


Abstract 


This paper describes the approach taken by Air Canada in 
carrying out a recent market research project leading up to the 
introduction of a new type of passenger service, and outlines the 
many factors considered. The author also covers the wide range 
of aspects included in the post-evaluation of this new service and 
of other on-going services. 
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Greetings everyone. It is a pleasure to have this opportunity 
to address people who are involved in the survey research field, 
and to share with you how we weave the survey research 
discipline into our marketing planning activities at Air Canada. 
We certainly recognize the need to be in touch with consumer 
needs and we consider consumer research to be an essential 
tool in the development of new products, refinements to existing 
ones and elimination of some which become passé. In other 
words ... in the overall marketing planning process. 


The Passenger Marketing Branch consists of four major 
functions: 


1. Strategy and Product Development 

2. Sales Development and Support; Product Launch Merchan- 
dising, Sales Acquisition and Distribution) 

3. Economics and Administration; Financial Evaluations and 

4. Advertising. 


The survey research function is within a larger group which 
includes forecasting, market research and the general dissemi- 
nation of information. This group is part of the Strategy and 
Product Development group, which is involved in the identifica- 
tion of market opportunities, the design, development, monitor- 
ing, and modification of our product. 


Now let me describe how a recent marketing program; 
Connaisseur Service; was conceived, developed and presented 
to our Sales force and operating people for delivery to the 
consumer. 


Connaisseur Service was designed to meet the needs of the 
full fare passenger, (the majority of which are travelling for 
business reasons) and many of the service features that make 
up this product were initially gathered from survey research we 
had conducted with the business traveller or acquired from 
sources such as: 


- Trish Hewitt and Assoc. Ltd. 
- Groupe Innova Inc. 


Our intention was to develop a quality product that provided 
the full fare passenger with a fast, efficient, trouble-free 
experience. 


Prior to the introduction of Connaisseur Service, full fare 
passengers received the same service as passengers on excur- 
sion and promotional fares. The objective was to improve the 
perception of value for money and satisfy the particular needs of 
this market segment. 


The individuals responsible for developing the product, and 
those involved in the implementation phase were assembled as 
a team which we identified as the Product Differentiation Task 
Force. 


The team’s objective was to consider ways of differentiating 
the existing product so the needs of the full fare passenger were 
met. 


Team members included marketing strategists, product plan- 
ners, ground and in-flight operations people, advertising person- 
nel, and marketing research specialists. 


Existing research on the needs of the business traveller was 
reviewed. One major research study which was designed to 
collect long and short haul business travellers’ reactions to new 
service features was particilarly useful. 


The methodology is not important to this discussion, however, 
| am sure the data was multivariated, standard deviated, sigma- 
two tested and conjoint analyzed. All of this might even have 
been massaged in a Jenkins Box. What impressed us was that 
the output was a simple ratio scale rank order of importance 
listing of the features. 


The list contained many features too numerous to review in 
detail here but I'll enumerate a few to illustrate. 


e Improved meal quality in business seating section which is 
separate from the discount fare section. 

¢« Seat selection at time of booking. 

e Issuance of boarding pass for the return flight and any 
connecting flights upon check-in at a Ticket Office or 
Airport. 

e Separate seating areas on aircraft for business and discount 
fare passengers. 

¢ Guaranteed choice of meals. 


| think my examples provide you with a good understanding of 
what we were trying to achieve -- provide the business traveller 
with as wide a range of features as possible. 


These service components became the framework for what 
was to become Connaisseur Service. Other complementary 
features which emerged from brainstorming sessions and the 
experience of the team were added and the product was 
finalized. 


The Strategy group prepared a paper which outlined the 
specific features which could be introduced without major 
modifications to the plant, and when they were to be introduced 
to the marketplace. The paper also included a financial summary 
of predicted revenues, costs, and an estimate of the impact on 
the marketplace. 
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While all of this activity proceeded, survey research was 
employed again on various name possibilities for the product. 
Connaissuer emerged as the label which best described the new 
service. 


Our advertising agency developed ads and these were pre- 
tested through the use of research techniques. 


The project moved to the Product Planning group which 
prepared a product brief which communicated the importance of 
key service aspects to those groups responsible for product 
launch, - the communicators -- Advertising, Public Affairs and 
Promotion, as well as the operations people -- airport, reser- 
vations and in-flight. 


The product brief identified the actual routes on which the 
service would be provided, and emphasized in more detail how 
it was to be positioned with reference to the overall objective of 
providing a high quality trouble-free product. 


The individual service elements were listed in detail and 
responsible groups for implementing them by specified dates 
were identified. 


Channels of distribution were listed and the advertising and 
promotion schedules were outlined. 


Connaisseur service was launched February 1, 1981. 


In April that year we conducted an in-flight survey to determine 
customer reaction to Connaisseur Service and gather informa- 
tion on travel behavior and demographics. We questioned 
passengers on: 


e how they became aware of the service. 

e the extent to which the existence of the service influenced 
the choice of Air Canada. 

e who actually purchased the product, where and when it was 
purchased. 

e duration of trip. 

e impressions of the individual service features. 

° how Connaisseur Service compared with the service pro- 
vided by other airlines. 

* purpose of trip, frequency of travel and, 

e demographics. 


The research indicated that the service was well received. 
Some problem areas were highlighted and corrective action was 
initiated. 


The service was expanded to many routes during 1981 and 
1982. A second customer reaction survey was conducted during 
April, 1982. More detailed questioning was included on the 
problem areas identified in the 1981 study. 


A more comprehensive line of questioning on how we were 
performing relative to the competition also produced actionable 
results. 


Thus in summary, we initiated four survey research programs 
to support the development of Connaisseur Service. These 
included: 


- a pre-test of product elements to determine consumer 
needs 


- aname preference survey 
- advertising copy testing and 
- two customer reaction surveys 


In a sense, one could say the requirement for Survey Re- 
search had been completed once Connaisseur had been introdu- 
ced. However, as we can all appreciate, the bringing of a product 
on-stream, does not mean there no longer is a need for research. 
In fact, the reverse is often the case. 


As | alluded to before, this introduction was merely phase one. 
We are still fine tuning the product. 


More refined phases are now in this implementation stage on 
our Atlantic services. One of its major components recognizes 
the need for more space and comfort on board the aircraft and 
to this end we are providing a separate cabin with (2x2x2) 
configuration instead of 9 or 10 seats abreast; that is to say, 
never more than two seats together. These are wider seats 
formerly offered in First-Class cabin, with greater leg room. 
Premium Class also offers china tray service, international 
newspapers and many other amenities formerly available only in 
First-Class. 


In recognition of the necessity to design the product to meet 
the demands of specific needs, a third phase of the program is 
in the development state for our North American routes. Survey 
research is being used in these phases as was the case in the 
initial one. 


Survey Research as it applied in establishing Connaisseur 
Service is one example of an Air Canada product. We have many 
products. Each one has been designed to appeal to a specific 
target audience ...... whether it be the discount traveller, or the 
Full Fare passenger. 


There are many important aspects that contribute to the 
development of a product ...... and, | have detailed some ...... but 
the one basic ...... the building block, if you will, remains constant 
Petisee research. 


Research is one of the integral components within our marke- 
ting equation. 


To leave Connaisseur Service and move to other applications 
of Survey Research, let me refer to our Customer Reaction To 
Air Canada Service, better known by its acronym label ...... 
CRACS. 


This is an inflight survey which we use to collect information 
on how our passengers rate service, their travel behaviour, how 
we rate against the competition, demographics, etc. It is conduc- 
ted quarterly and produces about 15 thousand completed 
questionnaires per quarter. 


- Passenger profiles by market area including trip purpose, 
booking patterns, frequency of travel and demographics to 
help product planners and advertisers focus on target mar- 
kets such as the business traveller, the price sensitive 
market, the female traveller, the high frequency traveller and 
geographic areas. 


- Advance purchase and duration of trip patterns to help in 
evaluating the effect of changing restrictions on discounted 
fares. 


- Our operations people use CRACS rating scale information 
to monitor our performance on specific service areas as our 
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customer sees it such as telephone reservations, on time 
performance, check-in and in-flight services. The rating scales 
measure the efficiency of the service and the performance of 
personnel. 


- Another recent Research project which was completed at Air 
Canada is the study of Future Travel and Transportation, a 
joint project which Air Canada initiated and which was 
co-sponsored. 


This was conducted in Canada and the US to determine: 


- the longer term future of travel and transportation in North 
America. 


- the competitive threats to travel and transportation as well as 
information on changing consumer needs and values. 


The results have provided us valuable insight for long-range 
marketing and airline planning concerning: 


* product planning, i.e., the types of products the business and 
pleasure markets are and will be interested in. 


* consumer confidence and the forecast of traffic growth. 
e emerging consumer destination desires. 
e the effect increased prices will have on the marketplace. 


e what problems other sectors of the industry are confronted 
with and competitive threats to air travel such as advanced 
communications and leisure activities which do not involve 
travel. 


e yet another research project is an examination of price- 
sensitivity with fliers. The objective here is to determine if the 
customer is prepared to trade-off features in favour of lower 
prices. This research provides us with valuable information 
on the discount passengers’ reactions to trading-off service 
elements and fare restrictions for lower prices. 


We have also conducted customized research in areas such 
as the likelihood of Business travellers purchasing more highly 
priced but higher quality products and at the opposite end of the 
scale on discount fares to determine the level of stimulation, - 
that is, new traffic to Air Canada versus diversion, - traffic moving 
down to the discount fare from higher fare types. 


The survey information is then merged with absolute traffic, 
revenues and costs to give us a net return. Survey research has 
also been used to: 


e track Air Canada’s image in the marketplace 
e develop information on advertising awareness 


e test advertising copy 
- on timetable formats 
- various types of wine 
- packaged tours 
- our ability to perform in Canada’s two official 
languages and our 
- automated ticketing service. 


As you can see; we use research for a variety of reasons and 
in a variety of ways. We do not limit ourselves to research which 
we conduct and develop; we also use the research of others; for 
example; we use the results of the Conference Board’s Survey 
of Consumers Buying Intentions, particularly the section on 
travel, and recently we subscribed to Tourscan, a service 
provided to the travel industry by L.J. d’Amore and Associates, 
one phase of which developed useful information on travel 
intentions. 


It is a great deal of research, | agree. It is costly ...... that is 
true. But it is also a necessary fact of the competitive and 
industrialized society. 


The survey research discipline has helped us at Air Canada in 
our planning and in our decision-making. It will continue to be an 
integral part of our Marketing process. But nothing, as the 
philosphers tell us, is perfect. The present is clouded by doubt 
and the future remains unclear. 


This uncertainty means ...... quite simply ...... that your role 
within the marketing equation will take on even greater 
importance. 


Survey Research, in a very real sense, is the stability factor. 
Your results provide that reality upon which so much is based. 


| have detailed for you the importance we place on Survey 
Research at Air Canada. The uncertain future, coupled with the 
requirement to re-act quickly to changing factors within the 
marketplace, means Survey Research will continue to play that 
integral role. | think the same is true of most major 
corporations. 


As aresult ...... you have a responsibility that cannot be taken 
lightly. From my own perspective, | can only ask that your 
profession provide the same dedication to fact and principles in 
the future, that it has shown in the past. 


Thank you. 
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PLANNING THE RESEARCH PROGRAM 


John Day, Vice-President, Marketing, 
H.J. Heinz Co. Ltd., 
Toronto, Ontario 


Abstract 


This paper describes how marketing research is considered 
an integral part of the marketing process at Heinz. Those factors 
which affect the sales of a particular product are identified for 
analysis, marketing objectives for the product are up-dated, then 
objectives are established for a marketing research strategy 
which will contribute to the development of a specific marketing 
plan. 


“Now, here, you see, it takes all the running you can do, to 
keep in the same place. If you want to get somewhere else, you 
must run twice as fast’. 


“Through the Looking Glass”’ 
Lewis Carroll, 1872. 


introduction 


Everyone is in some sense a marketer and is involved in using 
market research. For to market goods or services successfully 
over a period of time means that you must be in touch with your 
market. You must learn from it, test, analyze and adjust in order 
to progress. In fact, most Marketing planners are isolated in 
Company or Government offices far from their potential custom- 
ers and for this reason, we must rely on research to control our 
Marketing efforts. Marketing research, then, has become ‘‘a 
systematic process of information gathering and analysis for the 
purpose of improving decision-making and control in the market- 
ing of goods and services’. When viewed as a system of 
information, the purpose of marketing research is not only to 
solve specific problems but to provide continuous information 
which helps management to decide what problems to solve. 


This paper seeks to explore the indivisibility of marketing from 
marketing research within the context of planning and operating 
the marketing function at Heinz. A research program cannot be 
planned as an independent entity but only as a part of the total 
marketing plan, operations plan or budget. It is therefore, 
necessary to look first at the overall marketing plan, then to 
assess the role of consumer research within it, to examine the 
ways in which research assists in the development and control 
of the marketing process and the organization that best fits these 
needs. 


Heinz Canada has brand entries in a number of important 
consumer markets. | am going to take one of them and show, on 
a step-by-step basis, how consumer research is planned for it. 


Product Plan 
All activities in consumer market are guided by: 


(a) | Requirements of developing the annual Product Plan, 
and 

(b) |The execution of the marketing strategies, 

(c) | Monitoring/analyzing the success of the execution, 

(d) Preparation of next annual product plan. 


It is a continuous process of planning, testing, adjusting and 
re-planning. 


Role of Consumer Research 


Consumer Research plays an essential role in development of 
Product Plans, guidance in its implementation and monitoring its 
success. 


Specifically, the consumer research program is planned so as 
to: 


(a) |Help review canned pasta market, note important trends 

and developments. 

(b) Assess current positions of Heinz entries in different 

segments of market. 

(c) Understand problems, opportunities that exist for Heinz 

entries and possible new products. 

(d) Help develop effective strategies - product, packaging, 
pricing, advertising, consumer and trade promotions - this 
will maximize revenue/profits in market. 

) Help set specific objectives for these strategies. 

Monitor success of strategies as executed during year. 

) Be involved in analysis of success of strategies to prepare 

the next Product Plan. 


Qto 


Marketing is essentially future oriented and is concerned with 
projecting the future so that the corporation or organization can 
prepare for and cope with its environment over the next time 
span. Since no human can predict the future accurately, the 
process is one of continuously testing the environment or market 
place in order to enable the corporation to adapt to its market. As 
well as being reactive to changes, the corporation learns to use 
information to become proactive by seeking out opportunities 
which may be revealed in the analysis of information. 


Responsibility for Development of Product Plans 


At Heinz Canada responsibility for development of Product 
Plan is from the bottom up. 


Product Managers initially develop Product Plan, which is then 
discussed/changed/approved by successively higher manage- 
ment levels - Group Product Manager, Marketing Manager, 
within content of Product Plans developed for all consumer 
products in which Heinz Canada has entries. 


Plans of Consumer Research Program, as part of the total 
plan, follows this pattern. The initial responsibility is that of 
Product Managers, subject to discussion/changes/ approval of 
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progressively higher management levels. The market research 
plan is constructed and incorporated into the total product plan, 
using the following steps: 


e Product manager establishes on-going and ad hoc needs 
according to requirements of product plan. 
¢ Budget allocated on basis of experience/ advice. 
¢ Timing assigned to each ad hoc project to allow for: 
lead time discussion 
briefing of competitive research agencies 
execution of reporting 
e Execution of program according to the schedule established 
via use of research agencies. 


Coverage of Product Plans 


Let’s look at coverage of Product Plans, so we can see how 
consumer research programs need to be planned to meet its 
requirements in terms of development, implementation and 
monitoring. 


1. Business Review 
2. Problems and Opportunities 
3. Marketing Plans 
4. Financial 
sh Business Review 


Within any organization, there is a great body of knowledge 
built up through experience as well as special research projects. 
It is important that this information is kept up-to-date and is used. 


Problems 


(a) Market 
¢ Negative population trends 
¢ Competition from other foods in home 
¢ Competition from eating out 

(b) Heinz 


¢ Regionality of Sales Performance 
e Negative brand perceptions 
¢ Differing growth of market segments 


3. Marketing Plan 


The plan for the next year sets out the total marketing program 
in terms of the marketing mix: 


- Product 
- Packaging 
- Prices 


The purpose of reviewing the business as the first step in 
developing a plan is to ensure that it progresses from a firm, 
factual foundation. 


The business review is essentially backward looking and 
presents information and analysis of the competitive position of 
Heinz in terms of: 


e Market growth/decline 

e Size, growth, profile of consumers 

e« Sales volume 

e Brand share 

e« Consumer advertising, consumer and trade promotions 
e Product development 

e Pricing strategy 

* Consumer decisions, buying/consuming behaviour 

e Product/brand perceptions and attitudes 

e Interest in new products. 


The business review needs to be brief, but complete and if 
there are gaps in the knowledge base, they need to be filled by 
collecting the appropriate data. 


Pd. Problems/ Opportunities 


From the data base, we need to analyze and extract those 
areas where action is required. What problems and opportunities 
are revealed to us from the information so that we can address 
them in the marketing plan? 


Here are a few typical examples: 


Opportunities 


e Ad strategy to increase market growth, re-position 
brand, increase ad expenditures 

e Re-direct pricing policies 

e Consider food service+development 


e Re-distribution of marketing expenditures 
e Reformulate brand 
e Consider new products 


- Consumer advertising - Creative 

- Media 
- Consumer and trade promotions 
- Research plan 


The research program interacts with every facet of the 
marketing plan and could be detailed in this research program. 
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Review Area Timing Nature of research 
Market growth/brand share On-going Store audits (on-line access) 
Sales On-going Ex-factory shipments 
Advertising, promotion On-going Ad monitoring, Nielsen 
Consumers Ad Hoc National consumer surveys 

1 — 2 years 
Buying/consuming behaviour Ad Hoc Qualitative research 

1 — 2 years National Consumer surveys 
Brand perceptions, attitudes Ad Hoc Taste/in-home use tests, 

1 — 2 years Qualitative research, 

National consumer surveys 

New products Ad Hoc Major program of new product development 

as required 


In planning specific research projects, it is essential that the 
data be rigorously applied. To ensure that data is not collected 
simply as a matter of course, each project should answer to a 
specific set of objectives, which are always set before the 
research is conducted, preferably as part of the original market- 
ing plan. In this way, research can measure the progress of the 


marketing effort. Thus measurable objectives can be set for the 
brand in terms of market share, volume attainment, trial rates, 
re-purchase rates, income levels, etc., and for the component 
parts of the plan, research should be planned to provide informa- 
tion based on pre-set objectives such as: 


Strategies Objectives set Nature of monitoring research 

Product Ratings for reformulated product beat Taste, in-home use tests 
current 

Packaging Image ratings more in accord with Packaging tests 


brand positioning than current 


Prices Revenue maximization 


Creative Concepts and execution in accord with 


Regional tests 


Qualitative research, mail-tests, DAR test 


brand positioning better than norms 


Creative and media 
ratings and intent to purchase 


Consumer promotions Response rate 


Trade promotions 


Use of Consumer Research Agencies 


At Heinz, we rely on consumer research agencies to act as 
suppliers of information and they are asked to: 


e Draft recommendations to meet consumer research needs 

¢« Set action standards 

e Execute research 

° Report findings against action standards, recommend course 
of action 


Increase in brand awareness, image 


Match or better competitive brands 


Pre and post campaign tracking studies 


Comparison with response rates of previous 
consumer promotions 


Factory shipments, sales force reports 


Organization for Planning and Operating the Re- 
search Program 


Since we have viewed research as an integral part of the 
marketing process, it follows that the user of information must be 
the operating line marketing manager. At Heinz, we have no 
intermediary between the user (the product manager) and the 
supplier (the research agency) in the same way as the product 
manager relates directly to the sales force, the advertising 
agency, R & D departments, etc. This system has great strength 
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in making the user aware of the research process itself as well 
as the fact that information is an asset, with cost, timing and 


depreciation factors involved. 


Product manager needs following knowledge/experience: 


Research facilities 


Limitations of research 


Past research reference library 


Setting objectives, action standards 


How to evaluate research agencies 


Research reports 


The user of information is clearly not a technical specialist in 
producing data but must be capable of: 


¢ Defining his information needs 
¢ Implementing the information results 


By adapting to the market through a continous process of 
adjusting through feedback, the marketer plans to avoid the 
pitfalls of relying on intuition. Nevertheless research will never 
provide all the answers, to all of the questions, all of the time. It 
has been said that the recipe for a good decision is 90% 
information: 10% inspiration. 


This de-centralized system puts an onus on the users so that 
they need to know about: 


Range available 

New developments, especially computer capture, retrieval 
analysis of data 

Research agencies, personalities 


Strengths and weaknesses, particularly in area of predict- 
ing consumer behaviour 


Content 

Usefulness, problems to be avoided 

How new research can be standardized, add to past 
investment 


Avoid subjective evaluation 
More use of past research, norms 
Achieve rigorous research 


Strengths of research agencies, personalities 
Strengths, weaknesses of recommendations 
Price/quality relationship 


Assess results, analyses 
Check if action standards properly met 
Know how to implement results of research 


There have been tremendous strides in the techniques of 
market research in the past decade, but | see little evidence of 
a parallel improvement in the quality of the decisions that are 
made. As the computer has affected the supplier of research, so 
it will impact on the user in the future by giving him easier access 
to information, and the opportunity to manipulate the data to 
better quantify the effects of alternative courses of action. 
Nevertheless, the basic principles integrating the research plan 
within the total operating plan remain so that it is the quality of 
decision making and the conduct of the total planning process 
that is upgraded. 
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RESEARCH FOR PUBLIC AFFAIRS: A VIEWPOINT 
FROM A USER 
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Abstract 


This paper questions the practice of relying on competitive 
tendering (and selection of the lowest bidder) to obtain the best 
value for money when purchasing communications research. It 
suggests that better research would be obtained if a number of 
firms were paid to produce conceptual designs. One of these 
firms would be selected to develop detailed plans and specifica- 
tions. The operational aspects of the research could then be 
competitively tendered. 
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Mr. Chairman, it is a great privilege to be with you today. But, 
| want to confess that I’m really here under false pretences. Ina 
roomful of research experts from both the public and private 
sector, I'm almost as far as you can get from any research 
expertise -- but | do have some communications experience. In 
seven years in government and before that in the private sector, 
I’ve certainly commissioned a good deal of research. Yet, | have 
no deep specialist knowledge of your area -- much as | respect 
its vital importance in building good, effective communications. 
Nor am | expert either in the complex and delicate world of 
government contracting, which of course surrounds our whole 
topic. 


So while I’m a buyer all right, I’m certainly not a specialized 
purchasing agent. | buy survey research just as | buy newspaper 
space, radio and TV time, printing, artwork and all the other 
ingredients that go into communications programs. For many of 
these | do have access to expert buying, from DSS or my 
advertising agency. Let’s say then that | approach you from a 
position of experienced ignorance. And | don’t speak for the 
government or my organization -- the views | express are simply 
my own, based on a good deal of discussion with both research 
people and public affairs people. 


Research is, of course, absolutely central to effective commu- 
nications programming. Without it, the communicator and his or 
her clients are largly flying blind, relying on their own myths, 
shibboleths, prejudices, professional vanities and perhaps some 
skill. The ‘“‘seat-of-the-pants’’ approach is why much communi- 
cations money, public or private, may not be entirely well spent. 
Pre-testing, in particular, is essential to measure whether 
messages are efficient -- well-aimed and well-understood. It is 
not designed, as some cynics say, to measure whether the 
message is palatable. 


Do we always ask our customers and potential customers 
what they know or think about a problem, a product or a program 
before we begin to write? Do we always copy-test our material 
before we launch it, expensively, into that welter of communica- 
tions which surrounds us? And finally, do we effectively post-test 


our communication to find out how they were received -- whether 
the message got through or not? It’s really regrettable if we 
don’t, since the necessary research only costs a fraction of the 
total expenditure it’s designed to throw light on or measure. 


| don’t want to preach to the converted. !|’m sure we all 
welcome recent central directives that government advertising 
must be pre- and post-tested. (That pre-testing requirement was 
spelled out very recently in a Treasury Board circular.) Is this 
pre- and post-testing always done and how well? Also how much 
preliminary research is done to find out what the potential 
audience thinks about a policy or a program before any commu- 
nications are even drafted? (That is, basic communications 
research, rather than copy-testing.) There are two reasons why 
this basic research often isn’t done. First, people think they 
already know the answers and second, it’s harder to spring loose 
dollars for this sort of research than the more down-to-earth, 
bread-and-butter kind. However, we’ve certainly made consider- 
able progress in the amount and type of communications 
research that’s now being done in Ottawa. |’ve watched it grow 
enormously since | came here seven years ago -- from the 
pioneering work by the Canadian Government Office of Tourisn 
and what used to be Manpower and Immigration. But there us 
still plenty of room for progress. How much research supports 
those communications strategies we public affairs people now 
routinely prepare, for example? What percentage of all those 
publication we produce are being pre- and post-evaluated? 
When was the last time you heard of a publication being 
copy-tested? 


Within the recent past, the Treasury Board has admirably 
exempted advertising research from those extremely lengthy 
and time-consuming procedures considered necessary for ‘‘a 
related scientific activity’. They often acted as a deterrent to 
research. But for independent evaluations of advertising cam- 
paigns, DSS is the contracting authority, as it is for advertising 
itself. If DSS interprets its contracting mandate very strictly, there 
may be a strong impetus to use competitive tendering in pur- 
chasing that research, even though alternatives are possible. 


There is a growing requirement to include those millions of 
dollars Ottawa spends on public affairs in the ‘‘value for money”’ 
type of review so often mentioned by the former Auditor General. 
This trend can only de applauded. | believe that the better use 
of research can greatly aid that assessment -- it is a question of 
how we get better value for money in research to help us get 
better value for money in public affairs! 


My role today is to talk about the purchasing process for 
communications research, in company with a distinguished 
expert from the private sector. The perceptions | hope to share 
with you are, perhaps, those of many public affairs people, who 
want to research their work, want to test it, want to post-evaulate 
it. But |, at any rate, have some reservations about how best to 
buy this knowledge. | believe people from research houses in the 
audience have some reservations on how we try to do it, also. 


From my experience at buying communications research both 
here and in the private sector, | often have the uncomfortable 
feeling that we may not always necessarily be spending public 
money in quite the wisest way. In our proper zeal to reach what 
we perceive to be the lowest possible cost, the most economical 
product, we may be thinking solely of competitive tendering, as 
the only method. | know many people who do. If this is true, we 
may not always be getting the kind (or the quality) of research the 
government really needs. There is more than the possiblity that 


spon 


we are looking at the system the Government has developed for 
purchasing physical products as the only way of purchasing 
creative concepts and strategies -that is, the skills of people. (it 
would almost be more appropriate to use the government’s 
personnel selection procedures to hire research consultants. 
But, then, do we always hire the job applicant who will settle for 
the lowest salary?) It is wrong to believe that government 
policies absolutely require the selection of the lowest tender as 
a matter of course, but many people on both sides of the process 
seem to assume this. 


The contracting policy does, in fact, provide for selection by a 
number of other methods than low bids and by other methods 
than tendering. 


We as buyers and you as suppliers need to understand fully 
all the complexities and delicacies of government purchasing 
policies, and have a common view of them. We need, for 
example, to know that that policy permits what is called ‘‘pro- 
posal competitions’’, where we need to select a supplier on the 
basis of the level of performance offered, rather than on price 
alone. There is a whole procedure for this. My understanding is 
that, aside from certain rules, government contracts regulations 
do permit us reasonable freedom to negotitate a suitable con- 
tract with you to achieve our objectives as effectively as possible 
within the available budget. How to do its best is the question. 
We may all have gotten into the habit of thinking only of 
competitive tendering when other approaches are indeed possi- 
ble, permitted and, in fact, being used. 


At the same time, surely the research industry has some 
obligation to press the case for communications research more 
strongly. In these days of criticism of government communica- 
tions, particularly advertising, research has an important role in 
demonstrating that communication is professional, effective and 
well received. So our subject today has more than a technical 
interest. 


In this light, let’s look at tendering for research. 


Research houses, as | know them, usually wear at least two 
hats. They consist of professionals from a number of disciplines 
who offer a consultative service which appraises client research 
problems, designs solutions and then prepares the specifications 
for implementing the solution. (You could call this an architec- 
tural and engineering function.) At the same time, they’re 
contractors and sub-contractors in the operational business of 
printing and distributing questionnaires, stage-managing sam- 
ples and interviews, gathering and tabulating data and subjecting 
it to computer analysis. Then they interpret. 


I’ve deliberately just used some terms more familiar in the 
construction industry, because I'd like to make some compari- 
sons. If | want to build a building, pretty much the first thing 1’ll 
do after buying the land is to hire an architect. My architect will 
immediately be remunerated by an agreed fee-scale, to study my 
requirements, consult with me, do a sketch scheme and then 
work out the bugs with me. Then, with the aid of consulting 
engineers of various specialties, the architect designs the 
building to working drawing stage, backed by very detailed 
specifications. Only then are general contract tenders called. 


Notice that the competitive tendering process dosen't start 
until the construction industry has something specific it can 
price, more or less precisely. (I may have shopped around for the 
best architect, right at the start, but | certainly didn’t do it by 


competitive tendering! | talked to architectural firms about their 
concepts, viewed their past buildings, talked to their past clients. 
The hiring process may not have been competitive tendering in 
the usual sense of the phrase. But I’m sure the competing 
architects thought it competitive, just the same!) 


Sometime, several firms are paid to produce conceptual 
designs. The successful firm selected then goes on to develop 
detailed plans and specifications. Also design competitions are 
held. These too, are competitive processes but they are not 
competitive tendering. 


As far as | can see, very few professionals today are hired by 
competitive tendering, as such. That just isn’t the way they do 
their business. If | think I’ve committed a major libel, | wouldn’y 
ask Blake, Cassels and Graydon to be one of five law firms 
tendering on defending me. If I’m concerned about a complex 
financial problem which may be beyond our in-house expertise, 
| wouldn’t invite Price Waterhouse to bid on the job, along with 
six others. If | want to implement a management re-organization 
in my firm, | wouldn’t ask a group of seven firms, including 
Woods Gordon, to tender. 


Yet none of these professionals would say their business in 
non-competitive. They’d laugh hollowly. Nor am | suggesting 
that we hire professionals on a non-competitive basis. | am 
simply suggesting that we review the way we look as selecting 
them competitively. Perhaps paying them for proposals is the 
way to get their best thinking, for evaluation, for example. 


In each of these three analogies, the common ingredient is, 
perhaps, that our in-house skills don’t equip us to lay out exactly 
what it is we want tendered -- at least not in any degree of detail, 
nor with sufficient expertise. In the language of contruction, 
perhaps we need an architect before we can get competitive and 
comparable prices on the contracting work we may want done. 


Back, then, to research. In my experience, the core of the task 
in buying survey research well is to define the problem, solve it 
well and then lay out the operational procedures. In theory at 
least, when this has been done with precision, almost any 
competent, qualified contractor can carry out the operational 
work. Perhaps that aspect of the work can then be competitively 
tendered, by invited bidders carefully selected for reputation, 
experience and skill with that class of research work. Then you 
can get comparable competitive bids. But the key to making the 
research relevant, responsive and -- yes -- cost-effective, is that 
process that precedes it: the analysis of the client’s problem, the 
search for an approach to the solution and then the ‘‘working 
drawings and specifications’ for that solution, if you like. If that 
vital work isn’t done well, any amount of competitive tendering 
won't buy the right result. People will tender on the wrong 
product -- perhaps several of them. 


That joint immerision by the research house, the public affairs 
counsel and the client -- often these are the three corners of the 
triangle -- is the most important and sensitive part of buying 
research, in my opinion. 


(By the bye, | often think of the two U.S. space shuttle pilots, 
sitting at the controls of Columbia, blasting off into space when 
one says cheerily to the other: ‘‘Gee, it gives you a real warm 
feeling to know that every component in this crate was procured 
by competitive tendering from the lowest bidder.’’) 
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The Government of Canada, | suppose, would predictably 
always be more comfortable with research purchases by com- 
petitive tendering, in the name of fairness, prudence, protection 
of public funds, economy and efficiency. This approach is far 
more crediable, far easier to defend in Parliament and in public. 
It avoids any inacceptable excesses -- or any apparent excesses 
-- in the process. Yet are public funds always really being well 
protected by this process? For example, does a blanket request 
to 20 research firms (and without a very detailed specification) 
really produce even comparable proposals, never mind the right 
solutions? In a case like this, the decision may often largely be 
based on dollars, with less attention paid to the quality of thinking 
in each proposal, because the information given to bidders up 
front was vague. Thus the sketchy lowest-cost proposal which 
took less thought may be preferred. The thorough architect may 
be penalized for his/her deeper thinking and labelled expensive. 
| understand that government policy indicates that the number of 
firms asked should be decided with discretion -- three to five has 
been suggested, depending on the size of the research contract. 
What we seem to need is fewer bidders, better briefed and 
encouraged to show off their architectural skills. 


While Government policy permits a variety of purchasing 
approaches, the hard fact is that any Government purchasing is 
always potentially controversial and the method used must be 
defensible in public. (The media love stories of alleged Govern- 
ment purchasing boondogles, but when did you ever read a story 
about hundreds of millions of dollars worth of uneventful pur- 
chases properly and prudently made? Good news isn’t news!) 


The private sector is often held up as a model of purchasing 
efficiency, cost-effectiveness and value for money. Well, how do 
private sector firms buy survey research? | recently contacted 
the heads of a number of large research firms to get their views 
on the way the government buys research. | asked then what 
percentage of their private sector business was obtained by 
competitive tendering in the sense we've been discussing. They 
said the average was 10-20 percent. In their view, 80 per cent 
of private sector research may be negotiated. (That ought to give 
us pause, before we ask 20 more research houses to bid on our 
next study.) 


In public affairs, we can’t much operate on a competitive 
tendering basis ... and with good reason. Take advertising, as an 
example, because it has quite close parallels with research. 
Advertising agencies are selected by a process other than 
tendering -- and are asked to research a communications 
problem. In the private sector, at any rate, several invited 
agencies are quite often paid a fee to produce proposals on 
which they are then evaluated and one is selected to act. (In the 
private sector, too, they are also often remunerated by fee, 
credited against media and production commissions earned.) 
The agency proposes a media-creative solution which certainly 
isn't obtained by competitive tendering. True, the media continu- 
ally vie with one another in presenting their stories to the agency 
media director, who’s a highly specialized purchasing agent. He 
or she weighs a media mix appropriate to the particular problem. 
Media directors often prepare what you could call a specification 
from which to buy media and when it’s approved, they buy. They 
obtain the best rates, consistent with frequency and volume 
discounts. But it’s really a hand-tailored, custom package. The 
creative work isn’t competitively tendered; it’s wrought by hand 
to fit a particular need. Production work isn’t usually competi- 
tively tendered, either, though it may be bought, by experience, 
at the lowest acceptable cost from known suppliers. But certainly 


no advertising agency would agree that their business isn’t 
competitive. Like the lawyers, chartered accountant and man- 
agement consultants, they’d laugh hollowly. 


Take one other area of public affairs programming -- produc- 
ing a publication. Here, we conceive it and write it, prepare a 
graphic format, often in-house, but often with the aid of outside 
consultant. (The consultants would often not be selected by 
tender, but rather from past experience, ‘‘Standing-Offer proce- 
dures’’.) When all is said and done, we end up with approved 
finished artwork. It’s only at that point that we can reasonably 
ask DSS to obtain printing prices for us by competitive tendering. 
But notice that by that time, we’ve got a very precise specifica- 
tion on which to tender -- the artwork itself. At that point, 
competent, qualified printers with the right equipment can be 
selected to price th job. The desired purchases has been very 
Clearly defined. 


The point | guess I’m making is that I’m not in business to 
design and write precise specification for the kind of research we 
need. Not many departments have this in-house design capabil- 
ity. And perhaps most shouldn't seek it. They may not need it 
full-time nor could they necessarily attract research people of the 
senior professional calibre they have to have, when they need 
them. | should have though that people who are actually 
designing and carrying out reseach day in and day out are almost 
bound to be more experienced and more current with the 
state-of-the-art than all but the most expert in-house profes- 
sional advisors. 


As a public affairs person, | am competent, | hope, to tell a 
professional research person what | don’t know about an 
audience | want to address. | am competent to describe the kinds 
of messages | think | want to convey. | can describe the depths 
of my ignorance about how the proposed audience might react 
to those messages (and, indeed, to the general area of the 
subject-matter, as well). If | sound to you very uncertain at this 
stage of a communications project -- well, you’re right, | am. All 
| can say is | know pretty much what | don’t know. That’s 
precisely why | need research. And very, very often, | submit to 
you that calling in several research companies and asking them 
for a firm price tender on my ignorance just isn’t the answer. We 
still don’t have enough of a spec. And | don’t have the technical 
knowledge to write it in a way that could produce firm-price, 
comparable tenders from which | can defensibly select a 
winner. 


Even a very extensive and detailed bidders’ conference prior 
to tendering probably isn’t much of an answer. We all know that 
bidders’ conferences are not the place where research compa- 
nies normally like to tip their hands to their competitors by asking 
questions which reveal their thinking’ They tend to send observ- 
ers who listen. Separate one-on-one meetins may be more 
effective, but they can reduce the appearance of equal treatment 
for all bidders. 


What I really need is a seasoned research consultant who can 
sit down and pry out of me all my concerns, what | expect from 
the research, what | hope to learn, what | want to prove or 
disprove. Much of this communications research pretty well has 
to be somewhat custom work cookie-cutter solutions generally 
won't do. It has to be designed to solve a specific problem about 
a specific program with a specific audience. 
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Above all, the lead-times in public affairs are almost always 
too short. To some extent this may stem from bad planning. But 
clients want fast reaction from us on fast-breaking decisions. So 
often, you can’t just lift the last study you did off the shelf and 
re-tender it. Deadlines in communications generally militate 
against lengthy tendering processes, even if they can be consid- 
ered valid ways to procure research. (Pre-research delivered 
hours before the copy must be finalized isn’t a lot of use. 
Copy-testing findings which arrive right on media deadlines often 
don’t allow adjustments to be made well and only at great cost.) 
I'll frankly say again that in the public affairs area, we and our 
clients could somewhat improve our long-range planning and 
factor in the research process properly. But there will always be 
deadline pressures. Yet, if we were to follow the idea of 
competitive tendering for research literally in the typical commu- 
nications project, the research would just never get done. And 
frankly, | suspect that’s why research often isn’t done. The cry 
of ‘‘no time’’ can be an excuse for inaction in an area that is 
always potentially controversial. So adherence to a rigid tender- 
ing appraoch may actually hinder or eliminate that vital research 
which makes communications really communicate! Is that really 
saving the Government of Canada money, except in the sense 
that money was saved that should have been spent? Obviously, 
we’re really seeking a judicious use of the competitive process, 
yielding the best value in meeting the need. Best value may be 
complex to define, but we have to define it if we want to buy 
wisely. 


At the same time, I’m willing to concede that repetitive studies 
may certainly be prime prospects for competitive tendering. So 
when you sum it all up, in construction terms, | really do need an 
architect. 


Ideally, then, at the beginning of this hypothetical communica- 
tions project, I’d like to be able to lay out - 


e The broad research which will tell me about the enviroment 
into which I’m about to launch communications 


* Customer attitudes to my subject-matter area and the pro- 
gram, if they know about it 


e The specific copy-testing research I'll use once I’ve devel- 
oped the communications product 


¢ The post-evaluation research to match which will tell me how 
well (or how badly) my communications product actually 
fared in the marketplace. 


For this span of planning, quite clearly | need an architect for 
my research. Logically, | need an architect who can work with me 
through all phases of the project, meshing one phase with the 
next and using knowledge gained in one to plan the next. Should 
| really have to change research consultants (by competitive 
tendering?) from stage to stage? Am | equipped to bridge the gap 
between different stages and different consultants? To both 
questions: No. 


| haven’t had the temerity to bring you a bunch of easy 
solutions to my problems and yours. | don’t have them and 
anyway, | think the solutions ought to come from the research 
community -the in-house experts in Ottawa, where they exist 
and the central agencies which have the jurisdiction over 
contracting. But | really believe we need input also from two 
other key actors in the equation: the non-technical, non-expert 
buyers like me and the expert counsellors represented by my 
fellow-panelist who is about to give you her view. This dialogue 


could produce improvements and solutions, by better under- 
standing of Government contracting policy. At any rate, that’s my 
hope -- in shooting off my mouth. 


| have a deep gut feeling which may not prove palatable to all 
the experts in this room that | ought to hire research consultants 
on fee, to advise me on an ongoing basis. Acting as architects 
do, they’d design solutions to my problems, right down to 
detailed tender calls and specifications, and, through DSS, get in 
truly comparable bids and help us choose the lowest acceptable 
bidder. | believe that might best meet my needs and yet respect 
the principles of proper public purchasing. 


But if you follow the logic of that construction analogy through, 
I’d want the same consultants to supervise construction (i.e. the 
implementation of the study), and give me their professional 
advice at the end of the project (i.e. interpret the findings). It 
might be hard in that context for a reputable research house to 
be both consultant and contractor as well. You can answer that 
best. And | don’t know how happy Gallup would be to be 
supervised by Goldfarb as a consultant or how Frank Lacey of 
Contemporary would feel if denied the opportunity of implement- 
ing a study he’d designed. And what if the consultant selected is 
also the most experienced contractor in that particular line of 
work? 


Yet to my mind, these might be the bones of one solution. 
That ‘‘architectural’’ expertise really is the core of what | need to 
get communications research which truly gives me value for 
money. 


The other solution is, perhaps, to follow more closely the path 
of the private sector -to develop ongoing relationships with 
research houses who become familiar with my ‘‘products’’ and 
specialize in them. They would make proposals and then 
develop and cost the specific studies they recommend. It’s not 
competitive in the tendering sense, but if I’m watching perform- 
ance critically and other research houses are hovering around 
looking for business, it may in fact be just as competitive as 
research procurement in the private sector. And the results may 
match needs more closely. 


What | know is that if | follow the tendering concept very 
literally | may invest an enormous amount of my time in the 
process -- time | may hot have with expertise | may not have. 
Often, | may get proposals which aren’t comparable: oranges, 
lemons and pears. Then | can face very strong pressure to select 
the lowest bidder (it’s simpler, safer, less controversial), even if 
I’m uncomfortable with the firm and their proposal. To me, that 
approach to the competitive process doesn’t guarantee the 
Crown of Canada the best product, but perhaps, only the product 
which is apparently the lowest-priced. As a counterpoint to all of 
that, however, we must all remember that the Crown has to be 
seen to be purchasing prudently, fairly, competitively and eco- 
nomically -- yet still get value for money. In the glare of public 
scrutiny -- that’s quite a trick. Our counterparts in the private 
sector don’t have to conduct their purchasing under quite the 
same floodlights we do in Ottawa. 


There are other approaches. In the interest of buying better 
research, for example, | could instrcut my advertising agency to 
buy pre- and post-research from Research House X as if | was 
buying from them directly and (in reality) | become the client. | get 
the findings and tell the agency what | want done about them. 
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Professionally, there’s nothing wrong with this. But with the 
greatest respect to Agency Y and Research House X, even 
though | have the highest regard for their professional probity, 
there can always be the appearance that the agency is manufac- 
turing its own rationale and post-evaluation and sitting in 
judgement on its own work, even though we know this isn’t so, 
in fact. It dosen’t look completely right (or sound completely 
right), even as a necessary expedient towards the laudable goal 
of buying better research for better comunications at short 
deadlines. 


Another avenue to be explored is the establishment of central 
communications research expertise which could be made avail- 
able to all departments and agencies. Central architects, if you 
like. 


So there it is. | hope this viewpoint has been of some value. 
Like many of my colleagues, | suspect, I’m somewhat frustrated 
with our understanding of research purchasing procedures 
which many not always necessarily yield value for money. I'd 
hope that the central agencies might decide to work with the 
research industry and users to highlight the better and more 
streamlined procedures already available for buying research 
which would both protect Her Majesty’s purse and improve the 
quality of her communications at one and the same time. The 
delicate and complex government policies already provide for 


more than just straight competitive tendering. Perhaps as buyers 
and sellers we simply need to understand them better, with 
encouragement and explanation from the central agencies. In 
other words, the problem may be more one of our perception 
than the absence of procedures we can use to buy better 
research more effectively. 


In the crowed marketplace that is Canada, the competing 
voices of marketers daily raise such a din that if our messages 
aren’t aimed right on target, we'll hardly be heard at all. Both as 
a public servant and as a citizen, | happen to think it important 
what the Government of CAnada be heard clearly, for only when 
citizens know what Government is actually saying cany they 
really accurately assess it. 


Purchasing survey research sounds like a highly technical 
matter, as indeed it is. Government purchasing at best has to be 
complex and always open to public scrutiny, since it is always 
potentially controversial. It isn’t simple to meet the sometimes 
opposing interests of public probity, professional performance 
and public affairs pressures. But the effects of inappropriate 
research purchasing practice have rather serious implications for 
all of us, both as professionals and as citizens. What can we do 
together to improve our performance? 
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PURCHASING 
PERSPECTIVE 


RESEARCH A SUPPLIER’S 


Marilyn Sandler, President, 
Creative Research Group, 
Toronto, Ontario 


Abstract 


This paper outlines some of the major difficulties encountered 
by research suppliers, particularly in the area of communications 
with the client organization. Character sketches of various 
archtypes of research buyers are presented, along with a 
description of the obstacles to communications that each type 
generates. 
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You might well be wondering about the appropriateness of my 
talking to you about buying research. Indeed, you may even 
resent it. After all, | am in the supply business -- have been for 
more years than | care to remember -- and what do | really know 
about buying research? In fact, these very thoughts passed 
through my own mind. On reflection, however, | feel more 
comfortable with the subject. After all, through these many 
years, | have been involved in many research purchases, and at 
the very least, can offer you my perspective on them. 


| doubt that we will have very much difficulty in coming to an 
agreement as to what constitutes a ‘‘good buy”’ in research. Our 
problems are more likely to arise when we attempt to come to 
grips with how best to arrive at a satisfactory purchase. 


For the record, let me cite a few criteria that | feel are essential 
in order that research be classified as ‘‘good’’. 


1. The research must be pertinent. It must relate to the 
problem at hand and address the relevant issues. It mustn’t 
become obscured by ’’my, my results’’ -- answers to 
questions that are interesting but irrelevant. 


2. The research must be actionable. It must provide direction 
for a decision. Research reports that are destined to be dust 
collectors do us all in. And, 


3. It must provide good value for the money -- not only must 
the research be affordable within ‘‘whatever budget cons- 
traints exist’, its cost should be geared to the value of the 
decision to which it relates. A $60,000 commercial with a 
related media budget of $500,00 is worth more than 2 
group discussions: a package change of a minor nature is 
not worth an expenditure of $25,000 in packaging 
research. 


Prior to detailing the various means by which one might 
proceed to buy so-called ‘‘good’”’ research, there are a couple of 
other general comments that | would like to make. 


The first of these relates to the role of the buyer of research. 
And in this context | am referring basically to an individual whose 
main function in a company or organization is research -- 


whether it be in government, in a manufacturing organization, or 
in an advertising agency. All too often, these individuals see 
themselves simply as buyers -- people whose roles are to place 
the orders of their ‘‘clients’’ -- the ultimate research user -- in as 
wise a manner as possible, of course. 


This is, obviously, a key part of the job; it is not their sole 
function, however, and when it is treated as such, it does a 
disservice, not only to the individual personally, but to the 
organization involved and to research in general as well. 


The research buyer has at least two other responsibilities. 
One is relating to the outside world; making sure that he or she 
is in touch with and well-informed about the goings-on in the 
community. And, yes, keeping suppliers informed, happy and 
enthusiastic. There is little doubt that the enthusiastic supplier, 
in tune with the client’s problems and research needs, performs 
better. The research which stems from the type of symbiotic 
relationship which | am advocating is bound to be more pertinent 
and more useful. 


The other major role of the research buyer consists of relating 
to colleagues; keeping in touch with their plans and objectives 
and ‘‘selling’’ research to them -- selling, not with the objective 
of doing more projects and building a personal empire, but rather 
selling the general benefits and value of research as a marketing 
tool. And, in the process, of course; selling herself or himself. 


This is accomplished by making a contribution to the mar- 
keting or planning process, as the case may be. A paramount 
aim of the research buyer should be to cultivate a general 
enthusiasm for research among his or her colleagues, and he or 
she should take whatever steps are possible to promote the 
usefulness of research. The aim should be to ensure its role as 
an integral and essential ingredient in the organization. 


The research buyer who neglects these key aspects of his 
function is likely to find himself somewhat isolated from the rest 
of the world -- perhaps at best conducting outmoded research or 
at worst, and as has happened in this belt-tightening era, without 
a position altogether. 


The objective of research buyers, then, should be for a broad 
perspective; a perspective that understands and appreciates the 
value of research but that also extends beyond it to the utlimate 
planning process. 


To accomplish this, of course, requires a good deal of skill in 
the communication area. The process should look something 
like this: 


Sally Ursula 
Supplier User 


Boris 
Buyer 
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Unfortunately, it too often looks like this: Appropriate Action Standards which in turn help to ensure that: 


THE RIGHT QUESTIONS 
are asked of 
THE RIGHT PEOPLE. 


They will 
ENSURE MORE OBJECTIVE 
DECISION MAKING 
Sally Ursula and 
DISCOURAGE 
UNNECESSARY RESEARCH. 


Boris 


Or, even worse, like this: 


And, by the way, the answers to the questions | listed 
previously should be shared with all concerned -- not kept under 
lock and key for only a privileged few to share. 


For those of you who feel that these guidelines are universally 
applied and that research buyers who do otherwise disappeared 
with the Dark Ages, | thought it might be useful to outline a few 
character sketches of research buyers | have known (and loved). 
All the characters are real, but the names have been changed to 
protect the guilty. 


Sometimes, of course, this results in this: 


Boris 


Sally <——_________———__ Ursula 


And as you will all appreciate, this is not desirable from any 
point of view. 


To avoid this type of thing happening, to get off to a good start 
in any research undertaking, there are probably four basic 
questions which should be asked. Properly answered, the 
research direction should follow: 


These are: 
1. What is the objective of the research? For what purpose is 
research being contemplated? 
2. What is the current state of knowledge about the subject? 


3. What are the information gaps? 


4. What will be done with the information when it is Oliver Obstacle 
available? 


Obtaining the answers to these questions might involve a fair 


bit of ‘‘digging’’ -- both in secondary information sources and in Robe ee, ; ne A 
prior research documents. They will lead to: The first is Oliver Obstacle. Oliver’s favourite phrase is ‘‘that 


can’t be done’. This is his immediate response to almost any 

question. He rarely stops to think about how to solve a problem 
Better Problem Definiton (is it a problem that research can that has been posed, but spends much of his time deflating the 
solve?);and ideas that others might have. 
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How many times have the suppliers in the audience been 
approached by a research buyer who says ‘‘I’m sure this is not 
an appropriate subject for research, but those jerks in the 
marketing group won’t make a decision without it.’ One needn't 
work very hard to imagine the prior interaction between Oliver 
and ‘‘those jerks in the marketing department’’. | doubt that it 
was very positive or constructive. 


There are, of course, circumstances in which research is not 
appropriate. And it is very definitely the function of the research 
buyer to discourage unnecessary research. If this is the case, 
however, there probably should not be a call to a supplier at all. 
Or, if one seems useful, the need for research should be put 
forward with the request for a second opinion. It is not necessary 
to denigrate one’s colleagues; it only serves to emphasize the 
low status of one’s role, and places the supplier in something of 
an adversarial role in relation to his customer -- never a happy 
situation. 


Returning to Oliver for a moment, one gets the definite 
impression that Oliver is not happy in his job; he is always 
unenthusiastic and as a result, quite discouraging; rarely does 
one see him smile or laugh. In reality, Oliver is probably more of 
a problem for his colleagues and associates than for outside 
suppliers. In fact, he probably rarely gets to talk to the latter 
group. For his colleagues, however, Oliver represents a very real 
hurdle; to accomplish anything, they first must cope with Oliver. 
The result: little research is actually undertaken or research 
by-passes Oliver and he becomes redundant. 


Oliver or Oliver-clones are probably responsible for the fact 
that at least in some organizations, research, or the research 
department, is avoided at every opportunity. They simply make 
life too difficult. Of course, in establishing the objectives and 
determining the need for research, the research buyer may 
become involved in some uncomfortable exchanges with his 
colleagues. It is often beneficial that the researcher’s perspective 
-- potentially a very different one -- be presented. This is clearly 
important and should be granted a key role in the planning 
process. It should not be carried out with a negative and critical 
thrust however. 


Another character whom we’ve all come into contact with on 
one occasion or another is Harried Hannah. Hannah is so busy, 
so committed, that she rarely has time to communicate the 
rationale behind a study. Much of her day is spent telling all and 
sundry how very tied up she is; this explains why Hannah is 
generally late for meetings and why she has a tendency to cancel 
out at the last minute. Hannah also often doesn’t get around to 
requesting the research until a week or so before it must be 
completed. You know the type. A typical research brief from 
Hannah might consist of a telephone call saying ‘‘We want to test 
an ad’, leaving the listener to uncover -- hopefully prior to the 
actual execution of the research, the current state of the ad (Has 
it been running? Is it a pre-test?), and the existing concerns or 
feelings about it among those who are faced with decisions 
about the ad. In other words, the reasons for the research in the 
first instance remain a secret. The potential result, of course, is 
research that provides answers -- but to the wrong questions. 


| recently had a meeting which began some 30 minutes late 
-- because ‘‘Hannah’’ or Harry, in this case, was unexpectedly 
delayed. He informed me at the outset -- of the obvious -- that 
he was behind schedule -- that he would have to leave in 15 
minutes. The designated purpose of the meeting was to brief me 
so that we could prepare a proposal for a major research 
program which | understood to be very important to his compa- 
ny’s planning process. He actually stayed approximately 45 
minutes -- probably enough time to cover the subject reasonably 
well. The problem, however, is that we never did spend any more 
than about 15 minutes on the research. The rest of the time was 
spent discussing sports, restaurants around town and movies we 
had seen. Of course, he needed the proposal in two days -- 
during which time he would be out of town and out of reach for 
questions. 


As you can imagine, many assumptions had to be made; the 
risk is that when the research is executed, they may be accepted 
without question. And they might be wrong. 


Harried Hannah 


Egbert the Expert 
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Another individual we all know is Egbert the Expert. Egbert 
doesn’t bother with objectives. He specializes in design. A typical 
discussion between a research supplier and Egbert begins with 
‘“‘We want to do two group discussions’”’ or ‘‘We want a tracking 
study -- you know -- just ask the usual questions’’. A supplier, 
faced with this situation has two choices: he can accept Egbert’s 
dictum and do as instructed, or he can question. This can be a 
difficult decision, of course, since Egbert doesn’t often encoura- 
ge questions. It makes his life simpler if his directions are blindly 
followed and he communicates this quite emphatically. 


For example, | remember being called within the past year or 
so by a client who opened the conversation with something 
like: 


“We want to conduct a product test between 
our current formulation of our brand and 

a new formulation of the same brand. We 
think we have a better product.”’ 


This individual went on to indicate how large the product test 
should be, and stipulated that it was to be unidentified as to 
brand and was to be conducted in-home. 


What | learned only later,and with a great deal of probing on 
my part, was that the revised formulation of the brand was not, 
in fact, being considered as a better formlation of brand A, but 
rather was a cost improvement modification. The company was 
hoping to achieve taste parity, but at lower cost. This placed on 
entirely different complexion on the assignment. In fact, the 
design of the study was altered significantly by this piece of 
information. We actually conducted a taste test, as opposed to 
an in-home test; and we used a triangle test design, as opposed 
to a paired test. Moreover, the study was executed for a 
significantly lower budget than had we proceeded according to 
the original specifications. 


It is often extremely difficult, however, to elicit sufficient 
information from this kind of client. Such a person is often quite 
insecure and defensive; and it is rarely politic to try to dig out the 
truth from people at a higher level. 


Given Egbert’s modus operandi, it is quite possible that the 
final report and recommendations do not rélate to the problem. 
The reason -- it was not communicated in the first place. 


Agreeable Albert 


Agreeable Albert creates a different kind of problem. Albert 
always smiles and thinks every suggestion anyone makes is a 
good one. One difficulty with this is that Albert is always 
changing his mind -- depending upon whom he last spoke to -- 
and as you will appreciate, this can make research planning a 
Challenging exercise! 


Quincy Quicksand 


acy a 


Someone else who changes his mind a lot is Quincy Quick- 
sand, but he specializes in objectives. As the study goes on and 
the discussions progress, it becomes very difficult to pin Quincy 
down as to exactly why the research has been commissioned. 
And, once the results are in, there is often another alteration in 
Quincy's position. 


Action standards are another area which often seem to stand 
on shifting sands. And, of course, with each movement, there is 
something of a loss -- either in efficiency and potentially even in 
quality of interpretation. 
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Negative Nellie 


Negative Nellie is another favourite persona. Negative Nellie 
is in the habit of criticizing everyone around, particularly those in 
her own organization who are requesting the research. One gets 
the impression that everyone but Nellie is ‘‘out of step’. | wonder 
if she has ever stopped to think of the effect of this on the outside 
world. 


| can’t speak for everyone in this regard,but | can tell you that 
| personally find it very de-motivating. 


Then we have Vincent the Victim. Vince’s posture is to never 
assume personal responsibility for anything, anytime, ever. If he 
is making an unreasonable timing demand, it is never his request 
-- it is something he is being forced to do; if he is discussing a 
controversial subject -- or putting forward an idea which he 
anticipates might meet a different perspective -- it is presented 
as someone else’s suggestion. Vince is a fence sitter, his own 
position is never clear or communicated. Perhaps he doesn’t 
have one! Faced repeatedly with this situation, one is forced to 
come to the conclusion that Vince is either misrepresenting the 
facts on the one hand, or is completely without stature or 
influence himself on the other. Neither position is very satisfacto- 
ry, Clearly. 


Vincent the Victim 


The telephone shopper is another character that floats around 
the research scene from time to time. | recall taking calls on 
several occasions from individuals who ask what seems to be a 
very simple question. 


“How much will it cost to do a survey about ... whatever ...”” 


The subject doesn’t matter. It could be about anything. The 
point is, there is no more information forthcoming than the 
subject matter. | find it surprising that these individuals -- most 
of whom operate very succesfully in their own business -- and 
who wouldn’t consider offering their product or service in such 
an unprofessional manner, see nothing unusual about asking a 
question of this sort about research. It doesn’t speak very well 
for the opinion they hold about our profession. 


In all conscience, | cannot leave the subject of buying research 
without addressing the issue of competitive bidding. As a 
supplier, let me say at the outset that we do not, as you might 
imagine, have any inherent objection to participating in competiti- 
ve bids or in responding to competitive invitations. Although we 
might like the world to be otherwise, we are quite prepared to 
enter the fray when appropriate. It can and does offer a real 
challenge. 


The question, then, becomes ‘‘when does one generally enter 
a competitive situation with enthusiasm?”’ In simplistic terms, the 
answer to that question is ‘‘when one has a reasonable chance 
of success’’. And, of course, the more accurately one judges 
this, the better one’s track record. We cannot afford, either 
emotionally or financially, to participate in competitive bidding 
situations that do not hold out a good possibility of victory. It 
does not do justice -- either to ourselves or to our continuing and 
on-going clients. And, this is something all of us -- buyers as well 
as suppliers should bear in mind -- competitive proposals cost 
money, and ultimately they must be paid for. They should be 
asked for in those situations when they are justified -- when the 
number of dollars involved is very great; or when the research 
is fraught with very complex issues. 
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They should not be asked for situations where the decision 
has already been made. This wastes time and costs us all 
money. 


Recently, an invitation to tender was circulated with more than 
40 individuals or organizations listed; we declined to tender. I’m 
sure some other capable organizations did as well -- or at most, 
gave the project less than their total commitment. It would be 
difficult to do otherwise, under the circumstances. 


A possible result in this type of situation is that only those 
suppliers who are not as busy -- perhaps even those who are 
less qualified and less in demand -- will respond. To the extent 
that this occurs, -- and of course the difficulty lies in the fact that 
it is impossible to measure the extent to which it is a factor -- 
both the buyer and the ultimate user of the research suffer. 


With all due humility, | find it difficult to imagine a situation in 
which this number of bids has a constructive element to it. 
Surely, it makes more sense for all concerned to limit the number 
of ‘‘recipients’’ for any particular request, thus encouraging 
considered, pertinent responses from all. If there is a wish to 
utilize different suppliers for whatever reason, or at the very 
least, to give several suppliers an opportunity to participate, this 
could presumably be managed via different projects. 


| could, | imagine, go on ad infinitum, drawing my little 
character sketches about research buyers and in my own way, 
poking some fun. | haven’t mentioned: 


Stanley Staid - who never wants to try anything new. ‘“‘Let’s do 
it the way we did it last time. I’d feel 
more comfortable’. 


Suspicious Sarah - the research buyer who doesn’t trust sup- 
pliers and therefore doesn’t tell them 
very much -- particularly the back- 
ground and reasons why for the 
research. 


Celia Cram - who may be observed on most days, saying 
something like - ‘‘while we’re in the 
field, we may as well find out 
-da-da-do-da-da-da’’. Celia is behind 
much of the my, my research | men- 
tioned earlier. 


Thrifty Thomas - he always wants to do research in stages. 
“Let's do 25 interviews and see how 
it works out’, he often says. 


However, | probably have already overstepped some bounds 
and | can hear the retorts already. 


Who does she think she is: 


- Know-it-all Clara? 
or 
maybe even 


- Pompous Paula 
or, who knows? 


- Muddled Marilyn, maybe. 


Please be assured that | am very aware that there are at least 
as many characters sketches that can be drawn about research 
suppliers -- and they are undoubtedly equally justified. Unfortu- 
nately, however, you’ll have to wait until you’re asked to speak 
about ‘‘how to supply good research’ before you will be able to 
air them. | look forward to that time with some trepidation. 


Thank you for your attention. 
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METHODOLOGY OPTIONS 
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Managing Director, 
ABM Research Ltd., 
Toronto, Ontario. 


Abstract 


This paper reviews the range of options available to the survey 
researcher for data collection. Each is described and its main 
strengths and weaknesses discussed. The paper opens with a 
section considering the difference between qualitative and 
quantitative methods and, thereafter, the emphasis is on the 
latter, dealing in turn with face-to-face, telephone and self- 
completion interviewing. Then, cutting across this classification, 
space is devoted to panels and omnibus surveys. Finally, 
reference is made to observational techniques. 
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Qualitative or Quantitative 


In classifying data collection methods, the first split that should 
be made is that between qualitative and quantitative 
approaches. 


Qualitative research is small in scale, typically involving 
some 30-50 individuals; the total is sometimes less than this 
and is rarely as many as 100. While a programme of qualitative 
research will include a range of population sub-groups, defined 
demographically and/or behaviourally, respondents are not 
usually selected in any systematic, probabilistic way and the 
responsible practitioner makes nothing more than the mildest of 
claims as to their representativeness. 


Typically, the interviewer is also the individual upon whom the 
main burden of interpretation falls. The questioning techniques 
are relatively unstructured in nature and the overall shape of the 
interview might vary markedly from one respondent (or group) to 
the next, depending on the kind of responses that emerge and, 
therefore, on the avenue that the interviewer chooses subse- 
quently to follow. 


While hybrid approaches such as paired interviews and 
‘“‘mini-groups’’ can often be appropriate, qualitative research 
usually takes one of two forms: group discussions in which up to 
ten individuals are recruited to give a collective response at a 
session which is typically two hours in length and individual, 
‘in-depth’”’ interviews, one-on-one sessions typically lasting up 
to an hour. 


By definition, with a degree of luck in the recruiting and an 
adequately competent moderator, there is interaction between 
respondents in a group discussion. Comments made by one 
individual can trigger new thinking in another; argument between 
participants with opposing views can lead to an increased depth 
of understanding of consumer thinking; the way in which partici- 
pants change their views when exposed to the opinions of others 
can give insight into the dynamics of attitudes; a relaxed group 
setting can lead to a good degree of forthrightness. 


The decision as to whether to use group interviews or 
individual interviews (or, indeed, both) should depend on the 
extent to which this interaction is a hindrance or a help given the 
objectives of the study. For example, in the early stages of the 
development of an advertising campaign, the benefits of interac- 
tion between respondents are invaluable in giving direction to 
subsequent activity and in stimulating the flow of the agency’s 
creative juices. On the other hand, in attempting to gain some 
understanding of what is being communicated by an advertise- 
ment or commercial, interaction (and, therefore, the group 
discussion approach) is totally inappropriate. In a group, the 
comments made by individual participants will clearly be a 
function of the advertising plus the statements made by others; 
that would be fine if, in the real world, families and groups of 
friends were in the habit of setting up an impromptu group 
discussion whenever a new commercial is aired! 


Unfortunately, it is all too often the case that group discus- 
sions are undertaken rather than individual interviews either 
because the idea of using the latter simply does not occur or 
because the former approach is less expensive when costs are 
calculated on a per respondent basis. 


Quantitative research is fundamentally different from qualita- 
tive. It is larger in scale in terms of the numbers of respondents 
involved. In sampling respondents, attempts are made to ensure 
that they are representative of the population from which they are 
drawn. The interviewing is structured; that is, the interview 
follows a consistent, pre-determined course and individual 
questions are worded in precisely the same way at all 
interviews. 


In short, it follows from their different natures that: 


- qualitative research is concerned with the gaining of insights 
and impressions; 


- quantitative research is a form of measurement. 


However, although they serve different purposes, the two 
approaches are often complementary. For example: 
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Qualitative 


1. Hypothesis development 
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creative approaches, etc. 


3. An understanding as to: 


leading to 
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Quantitative 
Hypothesis testing 


A reduced and more manageable number 
for subsequent testing. 


A structured questionnaire which: 


a) all the issues, attitude dimensions —________________y a) is adequately comprehensive in its 
etc. which might be relevant to coverage 
the problem 
b) the approximate level of impor- 2S b) has the right emphases 
tance of a range of issues and 
areas of interest 
c) the kind of language used by a c) is optimally worded. 


consumers to express themselves. 


The flow can be in the other direction. For example, a small 
scale qualitative study, because of its greater depth, can often 
assist one in the task of interpreting a quantitative survey; i.e., 
the former can put ‘‘flesh’’ onto the ‘‘bones’’ provided by the 
latter. 


Quantitative Options in Summary 


There is a wide range of data collection options open to the 
quantitative researcher. These fall into two classes: 


- Personal, involving an interviewer in putting questions to 
respondents from a structured questionnaire; 


- Self-completion, where in the absence of an interviewer, the 
respondent completes the questionnaire himself. 


Personal interviews can, of course, be divided into two kinds, 
face-to-face and telephone, while face-to-face further divides 
between interviewing done in respondents’ own homes (or 
offices) and that done at a central location such as a shopping 
mall. 


Typically, self-completion surveys are mail surveys though, as 
we shall see later, a self-completion approach can be useful in 
other situations. 


In the following pages, each of these options is discussed in 
turn. 


Face-to-Face Interviews 


In terms of the number of dollars spent on market and opinion 
research fieldwork, the face-to-face interview accounts for the 
largest share and, generally speaking, forms the soundest basis 
for the collection of good quality data. 


The interviewer can take steps to ensure as relaxed an 
atmosphere as possible and she (or he) can proceed through the 
interview at a pace appropriate to each respondent. She is in a 
position to repeat a question when she detects that the respon- 
dent is misunderstanding something and she can pause when 
she sees his attention wandering. In particular, she can build up 
a good degree of rapport with the respondent, gaining a stronger 
commitment from him to the success of the survey than other 
methods allow. At open-ended questions she can probe for 
additional thoughts and for an elaboration of vague responses. 


However, it is possible for an interviewer to bias the results of 
a survey, not necessarily consciously, either by her manner or 
her appearance. For example, in a survey of eating habits, a 
team of overweight interviewers will get a different pattern of 
responses than an equally competent team of thin interviewers. 
Interviewer bias can be particularly serious when one is looking 
for regional differences in the results of a survey where only a 
handful of interviewers has been used in each region. In order to 
minimize the net effect of this bias, one should use as many 
(competent) interviewers as possible on each study. 


In-home interviewing can suffer from a relative lack of control. 
While the onus is on the designer of the study to keep things as 
simple as possible, sampling procedures and questionnaires are 
often necessarily complex and, even after thorough pre-testing 
and briefing, the field procedures can be error prone. It is usual 
to make a telephone callback on a proportion of each intervie- 
wer’s respondents, either via a local supervisor and/or from 
head office. Such callbacks provide a check on the basic honesty 
of the work and can establish that certain of the prescribed 
procedures were followed correctly. However, it is clear that they 
can never provide complete validation. 


A door-to-door interview is often impractical where it is 
necessary to expose respondents to an elaborate display, to a 
television commercial, to food prepared in a carefully controlled 
way, to a bulky product sample, to a prototype product which one 
has in very small numbers. In such situations, the interviewing 
must be done at a central location, for example by recruiting 
representatives of the relevant population to attend for interview 
at prearranged times. 


A more usual approach, unless one is dealing with a popula- 
tion sub-group accounting for a very small proportion of the total, 
is to intercept people in a shopping mail, to put the necessary 
eligibility screening questions to them and invite them immedia- 
tely to participate in the interview. 


Mall interviewing has the advantage of giving one a good deal 
of control over the work in that a supervisor is constantly in 
attendance. In some cases, this might be a sufficient reason for 
using such an approach but, in general, assuming that door- 
to-door interviewing is feasible, the latter is generally preferable 
(and is not usually significantly more expensive). 


Mall samples tend to be less representative in that, by 
definition, one is restricted to users of the selected mall(s); such 
people might not be representative of the wider population to 
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which one will subsequently be generalizing one’s findings. And, 
there is the additional potential source of bias that, amongst a 
mall's shoppers, an individual’s chances of being asked to 
participate are broadly proportionate to his/her frequency of 
using that mall. 


Further biases can be introduced by the fact that shoppers are 
very often in a hurry. Thus, busier people can be under- 
represented, a problem which increases in scale as interview 
length increases. When using an in-home approach, an appoint- 
ment can be made for a subsequent call if the selected respon- 
dent is busy. In a shopping mall, one has no such opportunity. 


Finally, it is relatively easy for a shopper to avoid a mall 
interview by bypassing ‘‘the lady with the clipboard’. It requires 
a greater effort of will to refuse to talk to an interviewer who 
knocks on one’s door at home. 


Thus, beyond the fact that a mall sample is restricted to users 
of that mall, there is a tendency for it to over-represent more 
frequent shoppers, people who are in less of a hurry and people 
with a stronger than average desire to be interviewed. Therefore, 
there is an unknown degree of atypicality in the sample, for which 
one can only partially correct by imposing quota controls on 
certain aspects of relevant demographics and behaviour. 


Many suppliers of mall interviewing services have perma- 
nently rented retail space for the actual interviewing while others 
use temporarily screened-off areas. AS a consequence, the 
environment for the interview is usually good in that distractions 
can be minimized, often to a much greater extent than is possible 
in home. 


However, interview length is an important issue. Given that 
one is attempting to stop people while shopping, 20 minutes is 
about the maximum. The present writer feels more comfortable 
with a study involving 50 minute in-home interviews than with 
one based on a 25 minute shopping mall interview. 


There is a large number of additional situations where central 
location personal interviewing is appropriate; i.e., where the 
population that one is sampling is itself centrally located. Exam- 
ples include in-flight surveys and airport intercept interviews 
amongst air travellers, on-site interviewing of exhibition atten- 
dees and store exit interviews with users of specific retail chains. 
In such situations one is, of course, restricted to a very short 
interview; however, additional information can be sought at a 
follow-up interview in-home or by telephone. 


These studies are in fact more complicated than they appear 
at first sight. Consider for example a survey of shoppers at a 
particular store, where one is conducting a short interview with 
customers as they leave. An individual’s chance of getting into 
the sample is proportionate to his frequency of visiting the store. 
Therefore, in principle, one’s sample is not representative of the 
store’s shoppers; rather, it is representative of shopping trips 
(which is usually a more appropriate base anyway). However, 
the sample can easily be converted from trips to shoppers if one 
remembers to ask those interviewed for their frequency of using 
the stgore. We then have a basis for weighting trips to shoppers 
by, for example, giving the ‘‘once a month’”’ respondent a weight 
of 4.0 relative to the ‘‘once a week’’ respondent. 


Second, one has to assume that the kinds of people who use 
the store at a particular time of day on a particular day of the 
week are different in relevant respects from those shopping there 


at another time and day. An implication of this, to ensure 
representativeness, is that one must organize the interviewing in 
such a way that its distribution across the week is proportionate 
to the total flow of shoppers across the week. Alternatively, one 
can count the flow of shoppers by time and day and weight one’s 
sample of interviews to the distribution indicated by those 
counts. If procedures of this kind are not adopted, the tendency 
is to over-represent shoppers at quieter times of the week (when 
one can interview a relatively large proportion of the total) 
relative to those using the store at busier times. 


Telephone Interviews 


The telephone is a less intimate medium than face-to-face 
contact and, for this reason, telephone interviewing is, in 
principle, usually less satisfactory than face-to-face 
interviewing. 


On the telephone, it is more difficult for the interviewer to 
establish her credentials and it is easier for the respondent to 
refuse. The interviewer is less able to build a degree of rapport 
with the respondent and there is more room for mis- 
communication. The interviewer does not have the same ability 
to detect that a question is being misunderstood or to detect 
when the respondent’s attention is wandering. 


There is less flexibility in the interview itself. For example, it is 
not possible to use any kind of visual prompting or stimulus, and 
open-ended questions tend to yield a less rich response because 
of the difficulty of coping with the pause while the interviewer 
makes the necessary verbatim record. 


Further, a telephone interview has to be relatively short. The 
value of the telephone must be seriously questioned when the 
interview length exceeds 30 minutes. 


However, there are a number of advantages to the telephone 
when the length and nature of the interview can be accommo- 
dated to the limitations of the medium. Perhaps the most 
important is cost. It is impossible to generalize about the 
relationship of telephone interviewing costs to the costs of 
in-home interviewing. The former are usually lower and the gap 
is wider when only a small proportion of the population qualifies 
for inclusion in the study. And, if one is sampling from a list of 
individuals widely scattered across a city, telephone interviewing 
is at a particularly strong advantage. 


More generally, while sampling considerations lie outside the 
scope of this paper, it should be noted that there is no cost 
penalty in allowing a telephone interview sample to be uncluste- 
red across a city. On the other hand, because of the increased 
travel time required between contacts, the costs of in-home 
interviewing increase as the amount of clustering decreases. 
This is an important consideration since sampling error increa- 
ses with the amount of clustering. 


The degree of control that can be exercised over the intervie- 
wing depends, of course, on whether it is being done from 
interviewers’ homes or from a central location. A majority of the 
telephone interviews conducted by the leading research sup- 
pliers are done in the latter context, the extra control providing 
a further advantage over in-home interviewing. 
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Indeed, computer assisted telephone interviewing, which is 
now appearing on the scene, takes this control a stage further. 
The next question appropriate to the respondent appears on a 
V.D.U. in front of the interviewer; she does not have to work out 
any skip patterns herself. And, certain edit checks can be 
programmed in at the outset so that inconsistencies in response 
can be flagged, and resolved, as they happen. 


Finally, telephone interviewing can be considerably faster than 
the door-to-door approach. In principle, if one’s budget allows 
one to accommodate the necessary long-distance charges, it is 
possible to undertake a national survey from one centre, without 
the need to allow time for large quantities of paper to travel to 
and fro across the country. 


Indeed, with computer assisted telephone interviewing, or with 
a keypunch operator on hand at the interviewing centre(s), the 
provision of tabulations can be virtually instantaneous. 


Mail Interviews 


As was pointed out earlier, self-completion interviewing repre- 
sents a further group of techniques, the most common of which 
is the mail survey. 


Assuming one has (or can readily establish) an unbiased, 
up-to-date list of the members of the population under study, the 
mail option should be considered. The main advantage of mail 
studies is, of course, their relatively low cost, particularly where 
the population is widely scattered geographically. 


The main deterrent to adopting a mail approach, however, is 
the often low response rate that it achieves, the response rate 
being the number of people returning a fully usable question- 
naire, expressed as a percentage of the total mailed. 


In principle, a low response rate is not a disadvantage if those 
who do respond are representative of the total. Unfortunately, 
that is not usually the case, in that respondents tend to be those 
with a greater than average degree of interest in the subject. 
Moreover, in practice, one does not usually know to what extent 
those who respond are typical or atypical since one only has 
information from respondents; nothing or very little is known 
about non-respondents and, therefore, one cannot estimate the 
bias introduced by the absence of data from them. 


Apart from the concern that they introduce as to the represen- 
tativeness of those who respond, non-respondents still have to 
be mailed in the first place. Therefore, they increase the cost per 
usable reply, a significant factor if the rate is low and/or if the 
mailing is an elaborate one, involving the inclusion of expensive 
stimulus material. 


Response rates also have the tedious characteristic of being 
very variable and, without thorough pre-testing, are difficult to 
predict in advance. The writer has experienced levels of below 
10% and above 90%. 


Significant, though not usually dramatic, increases in the 
response rate can be achieved by the use of such devices as 
reminder mailings for those who do not respond first time around 
and the inclusion or promise of some financial or other material 
incentive. 


Perhaps the biggest single factor is the quality of the question- 
naire and the persuasiveness of the covering letter. The respon- 
se rate will be enhanced if the questionnaire is clearly and 
attractively laid out, if it is interesting and if it includes the 
reasons for asking those questions of a more personal kind. If the 
nature of the research problem is likely not to be interesting to 
respondents, then a range of potentially interesting questions 
should be included, even if they are irrelevant; one doesn’t have 
to analyze them! 


Within limits, questionnaire length is not a factor. If it is 
interesting and professionally put together, a document requiring 
20 minutes’ work from respondents will not yield a response rate 
materially lower than one requiring five minutes’ effort. 


Response rates are strongly affected by the amount of prior 
contact that one has had with the sample. The level will be 
relatively high if the study is a follow-up amongst people who 
have already participated in an earlier study, particularly a 
door-to-door one. On the other hand, the level will always be 
unacceptably low if, for example, one writes ‘‘cold’’ to a sample 
of the population drawn from a telephone directory. 


There are ways in which one can to some extent correct for 
non-response bias. For example, if the sample mailed comprises 
people interviewed in an earlier study, those who respond can, 
on key variables, be weighted to conform to the ‘‘true’’ popula- 
tion profile as indicated by that earlier study. Alternatively, an 
attempt can be made to interview a sample of mail non- 
respondents by telephone; such telephone interviews will, to 
some extent, enable one to establish important respects in which 
the mail respondents are atypical and, thus, give a basis for 
weighting the latter. 


Assuming pre-testing has demonstrated that a mail approach 
is viable in a given situation, there are a number of important 
points to bear in mind about the actual completion of 
questionnaires. 


First, one must assume that respondents glance through the 
whole questionnaire before going back and starting at question 
1. Therefore, mail is inappropriate if later questions give the 
respondent information which you do not want him to have while 
he is answering earlier ones, and if it is impossible to include an 
adequate degree of camouflage. 


Second, the respondent has an opportunity to do a certain 
amount of research prior to completing the questionnaire. This 
rules out the inclusion of questions seeking levels of awareness 
and knowledge and is a disadvantage if one is seeking the 
respondent’s own opinions without any possibility of artificial 
“contamination’’ by other people. However, it is an advantage, 
where you are asking the respondent to report on behalf of the 
entire household or where the questionnaire requires him to 
check through the contents of his liquor cabinet, his inventory of 
footwear, entries on his bank statement, etc. 


Third, mail questionnaires can be the subject of a good deal 
of though (as distinct from research) before they are completed. 
Thus, the response tends to be a more considered and rational 
one than is obtained at a personal interview. Clearly, depending 
on the nature of the study, this can be an advantage or a 
disadvantage. 
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Other miscellaneous points to bear in mind include the danger 
that the wrong person will complete the questionnaire and the 
relative difficulty of getting a good depth of response if a large 
number of open-ended questions is included. On the other hand, 
the anonymity of the approach can lead to a more frank answer- 
ing of personal and sensitive question areas. 


Mail is, of course, the slowest approach to data collection, 
particularly if reminder waves are incorporated. 


Before leaving mail, reference should be made to mail panels. 
Several research suppliers in Canada operate such panels, 
which comprise several thousand households, nationally repre- 
sentive in terms of key geographic, demographic and socio- 
economic variables. They are households who have agreed to 
accept and return mail questionnaires on a continuing basis and 
representativeness is maintained by replacing households that 
fall by the wayside with others with similar characteristics. Such 
panels can be a very useful vehicle in that response rates for 
studies using them are very high (70%90%). However, it should 
be remembered that such response rates are to some extent 
illusory; while the sample is representative in terms of the 
variables by which it is deliberately balanced, it clearly cannot 
represent the large numbers of households who were unwilling 
to join the panel when invited to do so. 


The same problem arises when durables manufacturers use 
warranty cards as a sampling frame for mail studies. Even with 
a high response rate, those who did not return a warranty card 
in the first place can never be represented. 


Self-Completion Questionnaires, Other Than Mail 


As indicated earlier, not all self-completion questionnaires are 
mail questionnaires. A self-completion component can be useful 
within a personal interview, particularly when respondents are 
asked (perhaps more than once) to react to a long battery of 
attitude statements. Such an approach adds variety to a long 
interview, it can be quicker to administer (and therefore less 
expensive), and it can generate less irritation and fatigue than 
would repetitive verbal questioning. 


If a personal in-home interview is over-long, certain question 
areas can be included in a separate questionnaire left with the 
respondent for subsequent completion and pick-up by the 
interviewer. If certain questions require the perspective of other 
household members or if some research is required from the 
respondent, the placing of a self-completion questionnaire for 
later pick-up can again be highly appropriate. 


A further class of self-completion document is the ‘‘diary’’, 
used where one requires detailed, behavioural information which 
would not be adequately recalled at a single interview. Respon- 
dents are required to record, over an appropriate time period, 
behaviour as it happens; for example: media exposure, trips 
taken by various forms of transportation, consumption of food 
and beverages. 


If the collection of detailed information in areas such as these 
is crucial to the success of the study, it is likely that a diary will 
be the only viable option. 


In view of the fact that one is requiring a good deal of work 
from respondents, it can be difficult to recruit people and, once 
recruited, it can be difficult to secure the keeping of an ade- 
quately full record. A good deal of explanation and persua- 
siveness are necessary at the outset, in person, usually together 


with the promise of some material reward for full cooperation. If 
the diary is to be kept for more than a few days, intermediate 
contact with the respondent to spur him on is advisable. In order 
to ensure an adequately representative sample of participants, 
the setting of quotas on relevant demographic and behavioural 
variables is an important precaution at the placement stage. 
Needless to say, as with all self-completion documents, it is 
impossible to take too much troubie in ensuring that ease of 
completion is maximized. 


A final note of caution on diary studies: they usually lead to a 
large volume of data and the writer knows from bitter experience 
that it is all too easy to under-estimate data processing costs- -by 
a wide margin! Had Murphy been a survey researcher, he might 
well have stated that data processing costs on such studies are 
always at least twice as great as you thought they would be and 
at least four times as great as your client thought they should 
be. 


Panels 


In survey research, the term ‘‘panel’’ is used as a label for a 
sample of respondents who participate in studies on an intermit- 
tent or continuous basis over a period of time. 


Sometimes, the task that respondents are given varies over 
time. For example, reference was made earlier to mail panels; 
here, one of the options is to use respondents intermittently for 
once-off custom surveys. 


Similarly, consider the situation where one has a frequent 
need to sample a population that represents a very small, and 
therefore hard to find, proportion of the total. Having identified a 
number of them on one occasion, there are significant econo- 
mies to be gained by including the same pople in a future survey 
with quite different objectives. 


Alternatively, a panel can be set up in such a way that its 
members are asked to perform the same task over a period of 
time. Here the objective is usually to generate longitudinal data, 
giving one the opportunity of seeing how an individual’s beha- 
viour changes over time, for example as advertising campaigns 
come and go, as new product introductions occur, etc. Tracking 
studies where different samples of people are interviewed 
periodically only give one the ability to measure net changes in 
behaviour over time. The examination of gross changes (which 
collectively add to the net) requires a panel. 


With such panels one might be asking respondents to report 
continuously, for example by keeping a diary of their television 
viewing, of their grocery purchasing, etc. Alternatively, reporting 
might be intermittent: for example, a monthly telephone interview 
on behaviour in the preceding month (if the issues are ones 
where four week memory can be regarded as_ suitably 
reliable). 


The comments made earlier about recruitment for diary 
studies (which, of course, if reporting is continuous, become 
diary panels) apply with equal force here. Panel maintenance is 
also an important issue in that the usefulness of the approach 
depends on the same people remaining with you for an extended 
period; therefore, it is crucial that one be as persuasive as 
possible and that every opportunity is taken to build rapport and 
commitment. Insofar as ‘‘topping-up’’ is necessary from time to 
time to compensate for the wastage that does inevitably occur, 
the characteristics of those newly recruited should, in terms of 
key variables, match the profile of those who are lost. 
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An important potential source of error with panels is ‘‘condi- 
tioning’, the tendency for panel members to change in important 
respects as a direct consequence of their participation in the 
study. For this reason, panels should NEVER be used for the 
tracking of attitudes; once a series of attitude questions has been 
put to a respondent, one must assume that the way in which he 
views those issues in the future has been irrevocably 
changed. 


Indeed, the act of filling in even a behavioural diary can make 
respondents look at their behaviour in a new light and as a result 
lead to minor modifications of it. 


Omnibus Surveys 


In collecting survey research data, usually only a small 
proportion of the direct field cost is accounted for by time spent 
actually interviewing respondents. Collectively, briefing time, 
interviewer travel time and, in particular, time spent contacting 
respondents and establishing eligibility usually represent a much 
larger component. 


As a response to this situation, a number of omnibus surveys 
is available in Canada. Such surveys are ones where space in 
the interview is sold to a number of clients who pay on a per 
question basis. The value of such surveys is that clients who are 
participating in a given fieldwork cycle in effect share the field 
cost. 


Clearly, if you have a large number of questions to ask, you 
might just as well set up a custom study. Otherwise, because 
you are using most of the space available on the omnibus, you 
will be paying close to the full cost of the omnibus fieldwork. 


However, if you have a relatively short interview the econo- 
mies of sharing with others the set-up costs and the contacting 
and screening costs can be substantial. 


Further, if the population in which you are interested repre- 
sents a small proportion of the total, cost-effectiveness can be 
even greater. However, you then run the risk that the omnibus 
survey will include members of your population in insufficient 
numbers for your analysis. The solution here, if your timetable 
permits it, is to accumulate your sample over a number of 
fieldwork cycles, if the omnibus is one which runs 
continuously. 


Most omnibus surveys are based on door-to-door intervie- 
wing, varying in terms of size, fieldwork frequency and sample 
design. Mail and telephone omnibuses are also available. 
Omnibus surveys are not panels; a fresh sample of respondents 
is interviewed in each fieldwork cycle. 


Observation 


Hitherto, this paper has been concerned with techniques 
whereby respondents give information to the researcher by 
answering questions, either verbally or in writing. However, for 
completeness some reference should be made to observational 
approaches. 


Clearly, in areas where respondents tend to be unwilling or 
unable to verbalize an appropriate response, a lot can often be 
learned about the way in which they behave, and inferences can 
be made about the way in which they think, by watching them. 


Indeed, in any qualitative study, the researcher's eyes are as 
valuable a tool as his ears. 


How do people move round a supermarket? How do they 
decide which packages to take from a supermarket shelf? How 
do they cope with a new and different form of packaging? How 
do they cope with a new food preparation process? These 
questions, and many like them, can be illuminated by observing 
consumers in a systematic way. 


In recent years, advertising researchers have experimented 
with a number of techniques which involve the observation of 
physiological responses to commercials and print advertise- 
ments: psychogalvanomic skin response, eye movement, pupil 
dilation, voice pitch and even salivation. The advantage of these 
methods is that they lead to precise measures, using the 
appropriate hardware, and that they are objective. The reason 
why none has achieved wide acceptance is the, as yet, insur- 
mountable problem of understanding what the results actually 
mean in terms of the decisions that the planners and creators of 
advertising have to take. 


A more valuable hardware based observational approach is in 
the field of television audience research. In many parts of the 
world, meters are placed on television sets in a panel of homes 
to observe when the set is switched on and the channel to which 
it is tuned. 


Within the context of personal in-home interview studies, an 
observational component can be useful. For example, having 
asked respondents what brands they have in the home, a visual 
check of what is actually in the cupboard, when compared with 
what was said, can provide additional insights. 


A good deal of observational research is done at the retail 
level: the measurement of distribution levels, the auditing of 
sales through a panel of stores, display checks, the measure- 
ment of actual retail prices, etc. 


In asking a retailer about his role in a customer’s purchasing 
decision making, he might play down the issue of profit margin 
or commission. Additional insight might well be learned by 
having a second interviewer pose as an actual potential pur- 
chaser of a tennis raquet, refrigerator, package holiday or 
whatever, and asking her to note down as soon as she leaves 
the order in which brands, etc., were mentioned. 


Conclusion 


Therefore, a wide range of data collection techniques is 
available to the researcher, each with strengths and weaknesses 
and each making different budgetary and time-frame demands. 


In designing a study, the question is not necessarily that of 
deciding which one approach is best but, rather, is one of 
determining which mix of methods can together provide the 
optimum solution. 


Remember that the optimum solution is not necessarily the 
most elegant theoretically. It is the one which makes the best 
compromises by going furthest towards producing information 
which is complete and actionable within the twin constraints 
(always present) of time and money. 
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SAMPLE DESIGN: MAKING THE RIGHT CHOICES 


Allen R. Gower, Senior Methodologist, 

Census and Household Survey Methods Division, 
Statistics Canada, 

Ottawa, Ontario. 


Abstract 


This paper discusses the key decisions to be made in sample 
design. For survey objectives and data requirements to be met, 
it is necessary to make the right choices of population, frame, 
sampling method and sample size. 
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introduction 


My work with the Federal Statistical Activities Secretariat at 
Statistics Canada involves the review of the technical quality of 
surveys being carried out by federal government departments 
and agencies. In the course of reviewing the data collection plans 
for these surveys, we look at all aspects of the proposed survey 
methodology including sample design. We often find very good, 
sometimes innovative, sample designs being used. But it is not 
uncommon for us to observe that some of the survey research 
is being undertaken without enough attention being given to the 
basic principles of sample design in the planning stages of the 
survey, particularly with respect to the objectives and data 
requirements of the user. 


Designing a sample involves specifying the population, the 
frame, the sample selection method and the sample size. The 
choices and alternatives in the process of sample design are 
complex. The challenge that confronts us is how to put the 
known theoretical principles of sampling into practice and yet, at 
the same time, how to make the best possible use of available 
resources. 


Making the right choices in sample design will determine the 
success of the survey. In order that our objectives and data 
requirements are fully met, we must choose the right population, 
the right frame, the right sampling method and the right sample 
size. 


The Choices to be Made 


(a) What is the population to be surveyed? 


One of the first steps in designing a survey is to decide upon 
the ‘“‘population’’ that is going to be sampled. If the survey 
involves a study of consumer buying habits, will the population 
be households or people; or will the survey be targeted at males 
or females or a particular age group; or specific groups such as 
homeowners or car owners? The choice will depend, of course, 
on which population is most relevant to the objectives of the 
survey and the data requirements. 


Because the survey results will relate only to the population 
from which the sample is selected, it is essential to formulate a 
precise definition of the target population. Limitations in terms of 
type (households, individuals, businesses, institutions, etc.), 
geographic location, age, size and time frame should be stated 
explicitly. An appropriate target population for a survey to 
monitor automobile fuel consumption in Ontario might be ‘‘all 
automobiles registered in Ontario and which are driven during 
the reference month’’. The qualification ‘‘driven during the 
reference month’ is intended to exclude cars which do not 
contribute to fuel consumed because they were not used during 
the reference period. If the survey were focussed on automobiles 
driven for personal use, we could add an additional qualification, 
‘“‘automobiles having at least some personal use’’ to the defini- 
tion of the target population, in order to exclude vehicles such as 
taxis which are used solely for commercial purposes. 


Defining the population is not always straightforward. Consi- 
deration must be given to what is reasonable and practical. 
Although the target population should be identical with the 
population about which conclusions will be made, it is often 
necessary to eliminate certain segments of the target population 
so that we arrive at a definition for the population that is feasible 
to implement in the field. For instance, in a national survey, 
remote areas or northern regions of Canada are frequently 
excluded because they are too difficult or costly to enumerate. 
In the survey on automobile fuel consumption, we might have to 
consider the exclusion of remote regions of Ontario. Another 
practical decision might be to exclude automobiles sold to 
out-of-province residents during the reference month since these 
vehicles could be difficult to trace. 


Thus, for reasons of convenience and practicality, the survey 
population (i.e., the population actually covered by the survey) 
may not correspond exactly to the target population. The 
situation can be described as follows: the target population is 
determined by the survey objectives, but reality determines the 
survey population. When a survey population is chosen which 
differs from the target population, we must be aware that a gap 
exists between the two populations and recognize that conclu- 
sions based on the survey results apply only to the survey 
population. 


Once we have defined the population, the next problem that 
faces us is to decide on the most appropriate sampling unit. The 
sampling units that we use must cover the whole survey 
population with each member of the population belonging to only 
one sampling unit. 


Care must be taken in arriving at a precise definition of the 
sampling unit. Sometimes the sampling unit is the same as the 
population unit. In the fuel consumption survey where the 
population is of automobiles, the sampling units may also be 
defined as automobiles. On the other hand, in an employment 
survey where the population is of persons fifteen years of age 
and over, we may decide that a dwelling is the best coice for the 
sampling unit. A dwelling should be defined so that all persons 
in the survey population are covered and so that a person does 
not belong to two different dwellings. Statistics Canada, for 
example, defines a dwelling very precisely as ‘‘a structurally 
separate set of living quarters with a private entrance from 
outside or from a common hallway or stairway inside the 
building’. 
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The foregoing discussion and examples have illustrated that 
there are choices to be made when deciding on the survey 
population and sampling units. The right choice depends on the 
specific objectives and requirements of the study. 


(b) How do we find and locate the units in the 
population? 


Once the population has been defined we must find a means 
of accessing it. What we need is an appropriate frame (i.e., a 
complete list of sampling units covering the survey population). 
The frame should identify each sampling unit with certainty and 
enable it to be readily located. It should include names, ad- 
dresses and/or telephone numbers depending on the type of 
data collection method. The frame must be complete, accurate 
and up-to-date without duplication of entries or the presence of 
foreign elements in order to ensure adequate coverage of the 
population. It is helpful if the units on the frame are arranged in 
a manner which is suitable for sampling, if there is information 
about each unit (e.g., age, size, number of employees) which 
can be used to design an efficient sampling procedure, and if the 
frame is available in one central location. 


In some cases the choice of a frame may be easy. This is 
especially true when there is a suitable administrative file 
available. Consider, for example, a survey of benefit recipients to 
a government program such as Family Allowance. The list of 
recipients for this program should be complete and up-to-date 
since people immediately inform the authorities of births and 
address changes so that they can receive the benefits. 


On the other hand, there are many occasions when the choice 
of a suitable frame is difficult. What if the list is not up-to-date or 
has defects such as missing elements, duplication or errors? We 
have a variety of choices. We could use the existing list and 
ignore its defects. However, if we choose this alternative, we 
should inform users of the survey results of the existence of the 
defects and their potential impact on data quality. Another 
solution could be to discard the list, and use or create another 
list; or we could adjust the list by updating or linking it with 
another list in order to provide complete coverage of the 
population. However, creating or updating frames can be expen- 
sive and time-consuming, particularly if the frame is large or if the 
defects are difficult to detect. 


There are many kinds of populations for which no lists exist. 
For example, there is no list of all households in Canada, nor lists 
of men or women nor lists classified by variables such as age, 
occupation or income. During a recent survey of Canadian 
science teachers, one of the first problems encountered was that 
there was no list of science teachers. The solution was to obtain 
a list of all schools, use the schools as the sampling units and 
then enumerate all teachers who teach a science subject in the 
selected schools. 


In cases where a suitable or complete listing of the units in the 
population is not available or is expensive or time-consuming to 
compile, we may choose to use an area frame whose elements 
collectively comprise a list of geographic areas having well- 
defined boundaries. One of the best examples of an area frame 
is in the Census of Population and Housing where the frame 
consists of areal units called enumeration areas. Canada is 
divided into about 39,000 enumeration areas, each of which is 
the geographic area canvassed by one interviewer. Many survey 
research firms in the private sector also use enumeration areas 
as the basis for sample selection. 


Sometimes it is appropriate to use a conceptual list of popula- 
tion units when no physical list is available. Typical examples of 
this situation occur in surveys of airport users or park visitor 
users. In an airport user survey we usually choose to use a 
conceptual list of all persons using an airport facility (such as a 
departure lounge) during specified hours on selected days. 


The choice of the right frame is crucial to the success of a 
survey. | often maintain that the frame is the most critical element 
in the sample design of a survey. Without a suitable frame we 
may not get adequate coverage of the desired population which 
would mean that it would not be possible to generalize survey 
results reliably to the whole population. And, without complete 
addresses or telephone numbers the population units may not be 
able to be contacted. Nevertheless, ideal frames are almost 
nonexistent, and we must usually be willing to accept one or 
more imperfections which may result in over or undercoverage 
of the survey population. When choosing a frame for a survey, 
therefore, we should guard against imperfections. One way is to 
try and find out how and why the frame was originally created as 
well as how and when it was updated. This approach can help 
indicate the likelihood of defects occurring and how they might 
affect the sample design. But, even with this knowledge, we are 
often faced with the choice of using an imperfect frame or 
abandoning the survey entirely. In this case we must evaluate 
the extent of the frame’s defects and their potential impact on the 
survey results by re-examining the survey objectives and data 
requirements. 


(c) What sampling method should be used? 


The sampling method that we choose should provide a 
sample which is as representative as possible in the sense that 
it reflects the similarities and differences which exist in the 
population from which it is drawn. In order to have a sample 
which is representative of the population, sampling units should 
be selected without bias or, in other words, the selection process 
should be independent of human judgement. 


The type of sampling method which allows us to make 
scientifically valid inferences about the entire population which 
the sample represents is probability sampling. The distinguishing 
characteristics of probability sampling are that every unit of the 
population has a known, non-zero chance of being included in 
the sample and that the sample is drawn by some method of 
random selection. Probability sampling methods range from a 
simple random selection of the population units to very complex 
samples involving multiple stages of selection with stratification 
and/or clustering of the units into subgroups. 


The choice of which probability sampling technique is most 
suitable for a survey depends on the objectives and scope of that 
particular survey as well as considerations of cost and the data 
collection method. In a national survey to obtain the public’s 
viewpoints concerning a particular issue there are a variety of 
ways we could select the sample. If the survey were to involve 
a mail-out/mail-back questionnaire and estimates were required 
at the national level, a simple random sample (or systematic 
sample) might suffice. If we wanted data for every province 
separately, then we would choose a stratified sample design in 
order to ensure that all provinces are represented in the sample. 
If the information is to be collected by personal interview, then a 
single-stage sample would probably be out of the question 
because of the high costs and the problem of interviewing in 
widely dispersed locations. In this case we could use a two-stage 
design (either stratified or unstratified) where we would first 


om AT ie 


select a sample of areas and then choose a sample of respon- 
dents within the selected areas. Multi-stage sample designs of 
this sort are invariably used in personal interview surveys when 
the population is spread over a wide area. 


We often choose to use stratified sampling because it pro- 
duces a better representation of the survey population, because 
estimates are required for each stratum, or because it allows us 
to apply different selection procedures from one stratum to 
another. Stratification variables can be geographic (for example 
provincial, regional or urban/rural) or nongeographic (for exam- 
ple income, age or sex) and are the basis for forming homoge- 
neous subgroups of the population. Careful use of stratified 
sampling can improve the reliability and usefulness of the survey 
results and, at the same time, allows us to make efficient use of 
existing information and available resources. 


On the other hand, if we choose to use the cluster sampling 
technique with one or more stages of sampling, then we should 
recognize that this form of sampling will generally lead to an 
increase in the sampling error. This means that, if the margin of 
error is not to be increased, a larger sample size will be required. 
However, an increase in sample size will probably be more than 
compensated for by a lower per unit cost of enumeration. Where 
it is possible to choose the design of the final clusters, it is best 
to form the clusters so that the units which make up any one 
cluster are as dissimilar as possible with respect to the charac- 
teristics being measured. Further, since units within a cluster 
tend to be similar, it is generally better to choose to use a large 
number of small clusters than a small number of large 
clusters. 


Thus far | have focussed my discussion on probability sam- 
pling methods. The other main category of sampling methods is 
nonprobability sampling and includes methods such as judge- 
ment sampling and quota sampling. Nonprobability sampling 
methods are generally cheaper and more convenient to carry out 
than probability sampling methods because fieldwork can be 
carried out quickly by avoiding tasks such as listing and the 
follow-up of nonrespondents. However, with nonprobability 
sampling, it is not possible to estimate the probability of any one 
unit being included in the sample since units are chosen in an 
arbitrary manner. Consequently, results cannot be generalized 
to the whole population nor can sampling errors be estimated as 
can be done when probability sampling methods are used. 
Furthermore, there is no assurance that every unit in the 
population has a chance to be selected, and a biased and 
unrepresentative sample may result. 


This is not to say that we should avoid choosing a nonprobabi- 
lity sampling method. Indeed there are many situations where 
nonprobability sampling is the right choice. It is particularly useful 
in exploratory studies such as pretests, pilot studies and focus 
groups. There are many instances when nonprobability sampling 
is the only practical method of sampling; for example, if there is 
no frame available. Thus, we see many survey researchers 
making use of a modified probability sampling scheme in surveys 
of the general population. This generally takes the form of a 
stratified multi-stage sample design with the imposition of quotas 
at the last stage of selection where a suitable frame is not 
available. In this type of approach to sampling, interviewers are 
instructed to obtain a specified number of interviews in selected 
geographic areas according to variables such as age, sex, 
occupation or income. The choice of which variables should be 
used for quota controls depends on the objectives of the survey 
and on what data are available for the population at large. 


Although our aim should be to make the sample as representati- 
ve as possible with respect to some known characteristics, we 
should recognize that, since the method of selection is not 
entirely random, we are very much open to the danger of getting 
a sample that is not representative with respect to other key 
characteristics such as activities, attitudes and opinions. 


A recent survey of Canadian air travellers provides a good 
illustration of how the choice of sampling technique is very much 
dependent on practical considerations. Originally it was planned 
to have a carefully designed probability sample involving a first 
stage selection of flight segments and a second stage selection 
of passengers on these selected flights, with stratification by day 
of week and according to short and long haul flights. This 
sampling strategy had to be abandoned due to the fact that one 
of the questions asked of respondents required them to identify 
the fare type on which they were travelling. This could not be 
done without the aid of an interviewer who could interpret the 
complicated fare codes and price combinations on a respon- 
dent’s airline ticket. Since it was not feasible to have an on-board 
interviewer nor to have flight attendants perform the task, it was 
decided to use a quota sample of passengers in departure 
lounges. This sampling approach, although it was easy to carry 
out and allowed accurate identification of the fare structure under 
which a passenger was travelling, limited the generalizations 
which could be made from the sample results. Being a nonpro- 
bability sample, conclusions could theoretically be drawn only for 
the passengers interviewed. This, of course, imposed a serious 
limitation on the usefulness of the results and the findings of the 
survey. Yet it was the only reasonable way the required informa- 
tion could be collected. And, what was very important in this 
case, all users of the data agreed before the survey was 
conducted to recognize the limitations imposed upon the data 
because of the use of quota sampling. Thus, this survey 
illustrates a situation where the practical choice was to use a 
form of nonprobability sampling. 


Unless we want to draw conclusions only about the sample 
itself (in which case almost any kind of sample can be chosen), 
we should try wherever possible to use a selection method 
based on probability sampling techniques in order to get a 
sample which is representative of the population from which it is 
drawn. We are then able to generalize our results to the whole 
population. But, in general, the sampling method that is the right 
choice depends completely on our specific survey objectives and 
data requirements as well as what is feasible to implement. 


(d) How big should the sample be? 


1,000? 400? 100? 1% of the total population? 1 in 5? 1? As 
a survey statistician, the question of ‘‘how big should the sample 
be?’’ is probably the question that | am asked most often. The 
textbook answer is to decide on what margin of error is tolerable 
and, using whatever data are available on the variability of the 
population to be studied, to calculate the sample size that is 
necessary to keep within this margin of error. But, in practice, the 
choice of sample size is not so direct. Since every survey is 
different and has its own unique features, the determination of 
sample size requires a great deal of information about the survey 
and about the population from which the sample is to be 
selected. We must consider the desired precision of the survey 
results as well as cost, time and operational constraints. 


Because a sample represents only part of the population being 
studied, the survey results will be subject to some error or 
uncertainty. The larger the sample that is chosen, the more 
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precise will be the sample estimates (apart from any biases or 
errors which may arise independent of sample size). Therefore, 
one of the key decisions we have to make when designing a 
sample survey is to decide what margin of error we are willing to 
tolerate. This involves a close examination of our data require- 
ments including estimates of proportions, means and totals as 
well as planned cross-tabulations and hypothesis tests. 


In determining our requirements for reliability, not only should 
we consider data requirements at a global level, but we should 
also take into account any subpopulations for which we want 
separate results. Each subgroup should be large enough so that 
its survey results can be analyzed reliably on their own. A 
national survey, for example, may have a sample size of 1,000. 
For analyses carried out on the entire sample, the confidence 
interval (within + 0.03 at the 95% level of confidence for an 
estimated proportion of 0.50, assuming simple random sam- 
pling) is likely to be very acceptable. The situation can change 
quite dramatically if we want to analyze subgroups of the total 
sample such as age/sex groups or geographic areas. This is 
because the standard error of an estimate in a particular 
subgroup depends on the size of the sample_in that subgroup. 
We see, for example, that in a subgroup with a sample size of 
100, the confidence interval increases to + 0.10 for an esti- 
mated proportion of 0.50. Thus, the more subgroups for which 
survey results are required, the larger will be the sample size 
required to maintain the sampling error within acceptable limits. 
This is why a survey like the Canadian Labour Force Survey has 
a sample of 55,000 households (or about 120,000 individuals) 
every month. The multitude of breakdowns (by province, eco- 
nomic region, city, age, sex, industry, occupation, etc.) which are 
required together with the very tight requirements for data 
reliability demand such a huge sample size. At the other ex- 
treme, we know how often a sample size of 1,000 or 2,000 
people is used in national public opinion surveys. 


Once we have decided on the margin of error we are willing 
to tolerate, a number of factors influence the sample size which 
is necessary to achieve this level of precision. These are the size 
of the population, the variability of characteristics in the popula- 
tion, the method of sampling, and the degree of nonresponse. 


For most surveys, population size is not really very important 
in determining sample size. Although it plays an important role 
for small populations, it plays almost a nonexistent role for large 
populations. For small populations, due to the substantial 
proportion of the population which must be sampled to achieve 
the desired precision, the sampling fraction may be high enough 
to warrant taking a 100% sample (i.e., a census). This approach 
is often desirable in terms of cost effectiveness since we are able 
to save time and money by avoiding the process of sample 
selection. 


The variability of characteristics of interest in the population 
impacts on the sample size required since the greater the 
difference between units in the population, the larger will be the 
sample size needed to achieve the specified level of reliability. In 
a population where there is no variability at all, a sample size of 
1 would suffice. Unfortunately, the variability of population units 
is generally not known in advance. If possible, we usually choose 
to approximate it based on the results of pilot tests or previous 
surveys. Otherwise, we consider the case when maximum 
variability (e.g., a 50/50 split in a public opinion poll) occurs and 
use this to determine sample size. 


Nearly every survey requires information on a number of 
different variables. A sample which is large enough for one 
variable may not be sufficiently large for another variable 
requiring greater precision or, even if the same precision is 
required for all variables, a larger sample size may be necessary 
to achieve the same level of reliability for items having a higher 
degree of variability. How do we deal with a problem of this sort? 
We could choose a sample size which would be large enough to 
give the desired precision for all the variables. Otherwise, we 
may have to re-examine our objectives and data requirements 
and ensure that a satisfactory level of precision is obtained for 
those which are the most important. 


Also affecting the reliability of survey results is the method of 
sampling that we have chosen. While the standard approach to 
sample size calculation assumes simple random sampling, this 
may be inadequate in practice especially if a more complex 
design such as multi-stage sampling is used. Because a multi- 
stage design generally results in higher sampling variance, we 
should inflate the sample size in order to obtain the same level 
of precision as would have been obtained using simple random 
sampling. The inflation factor can be determined from other 
surveys of the same or a similar design. More often than not, 
however, we estimate it in a very subjective manner by using 
whatever knowledge we may have, even if it’s guesswork, about 
the variability of characteristics of interest in the population. 


The effect of nonresponse on sample size also needs to be 
considered. In any survey nonresponse is virtually certain to 
occur and, because of the reduced sample size, the margin of 
error will be larger than desired. This situation can be avoided by 
taking a larger initial sample size based on an expected rate of 
nonresponse (estimated from similar surveys or pilot test re- 
sults). While compensating for nonresponse in this way can be 
effective in reducing the sampling variability and ensuring the 
desired precision of the survey results, we must not forget that 
the bias resulting from nonresponse is not reduced. It may be 
that, with respect to characteristics of interest in the survey, the 
nonrespondents are different than the respondents. 


Up to this point we have looked at only one aspect of the 
choice of what size the sample should be: what sample size is 
necessary to achieve a specified level of precision? In practice, 
however, our decision on sample size is influenced just as much 
by the amount of time and money available as by the desired 
precision of sample estimates. We are also forced to consider 
how big a sample size we can afford in light of any constraints 
imposed by the availability of field staff, coding and editing staff 
as well as processing facilities. And it may happen, therefore, 
that the desired sample size in terms of data reliability, may not 
be possible to achieve due to cost, time and operational restric- 
tions. A balance must be reached between the value of in- 
creased information and the costs of gathering the data. The 
feasible course of action may be to take the largest sample which 
is financially possible and to reduce our data requirements, or 
even to omit items of interest or sample breakdowns for which 
a much larger sample is needed to give useful results. This 
decision requires a re-examination of our objectives and survey 
requirements, and it involves a process of making practical 
choices and trade-offs among competing factors. The right 
sample size will be the one that best meets all our conflicting 
requirements of precision, cost, timeliness and operational 
feasibility. 
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Concluding Remarks 


Making the right choices of population, frame, sampling 
method and sample size is critical to the success of a sample 
survey. If our choices are right, then we'll be satisfied with the 
results. But, if our choices are wrong, then we may find oursel- 
ves with results which are limited in their usefulness or even 
useless. So, let’s make the right choices at the outset when we 
design the sample. We can do this by paying very close attention 
to the purposes of our survey, our specific data requirements as 
well as how reliable we want the results to be, and by balancing 
what is theoretically desirable against what is practically 
feasible. 
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ADVANCEMENTS IN DATA COLLECTION - 
COMPUTER-ASSISTED INTERVIEWING 


Gill Humphreys, 
Vice-President, 
Canadian Facts 
Toronto, Ontario 


From the very beginning of survey research, until fairly 
recently, there have been no advancements in data collection. 


As this is designated as a workshop session, | thought | might 
throw in that controversial opener then sit down and let you all 
prove me totally wrong! 


However, before | do that, let me.explain what | mean. | am, 
admittedly, addressing myself to the simple, straightforward 
action (having found a willing qualified respondent by whatever 
appropriate interviewing method) of collecting and recording the 
information from a respondent. A small part of the survey 
research process, maybe, but the cornerstone of every 
project. 


The only tools available for this task have been the tried and 
trusted paper and pencil, operated by either the interviewer or 
the respondent himself/herself. Over the years, as the survey 
research business has blossomed and questionnaires have 
become longer and longer, | am frequently surprised that we 
have not all been taken to task by conservationists for the trees 
that have been sacrificed in our pursuit of more and more 
information! But now the wonders of science have caught up 
with us in the form of computer-assisted interviewing. With these 
systems, the questionnaire is programmed for implementation. 
The interviewer works with a cathode ray tube screen that 
displays the exact wording of the question to be asked. On most 
systems, the interviewer uses a typewriter-like keyboard to 
record the respondent's answer, which is then fed directly into 
a computer in machine language ready for processing. Depend- 
ing on the respondent’s answer, the computer then displays the 
next question on the screen, and so on. 


These systems are currently in use in the States and, to a 
more limited extent, in Canada, mainly in association with 
telephone interviewing from central locations for the obvious 
reason of logistics. 


There are a number of general operational benefits in com- 
puter-assisted interviewing which | would like to discuss before 
talking about specific applications in telephone interviewing. 


My talk is by no means an exhaustive review but will, 
hopefully, spark any thoughts, concerns or questions that you 
might wish to raise. 


The major overall benefit, | believe, from an operational 
viewpoint, is that we can simplify the tasks that we set for our 
interviewers. This will inevitably lead to better quality data. 


As information requirements have become more and more 
sophisticated and our data analysis skills have developed, 
enabling us to do complex data manipulation comparatively 
simply, the burden that we have placed on the data collection 
function has been phenomenal. 


| mentioned earlier that questionnaires have become longer 
but, also, they have become more complicated to administer. 
Computer-assisted interviewing definitely has the potential to 
ease these complications because many of the tasks, currently 
undertaken by the interviewer, are removed, allowing him or her 
freedom to establish a better rapport with the respondent. For 
example: 


- Skip patterns are automatically controlled, making handling 
easier and more accurate. 


- Rotations of all sorts can be included with no effort. 


- Split samples and split questionnaires are simple to 
execute. 


- Constant sums are automatically checked for accuracy. 


- Any series of numbers that has to add to a specific total 
(frequencies, percentages, etc.) is handled more easily and 
accurately. 


- Loops (that is, when a series of questions is to be asked 
more than once) are easier to administer. 


- The facility to set up response lists and sub-lists means that 
the interviewer only has to handle items that are relevant to 
the particular respondent being interviewed at that time. For 
instance, if the respondent buys Brand A of yoghurt, which 
is only available in specific flavours, only those flavours will 
be displayed in subsequent questions about usage of Brand 
A. 


All of these items are examples of how a questionnaire can be 
made easier to administer. As a side benefit, the need to design 
questionnaires that interviewers can cope with is less of a 
concern -- although | have sometimes wondered whether this 
has ever been a concern for some questionnaire designers! 


Now, let us look at some of the benefits which are specific to 
the telephone interviewing mode. Here, there are some aspects 
which help the interviewer and, also, many which assist the 
research supplier in terms of control and monitoring. 


Some of the important features of computer-assisted tele- 
phone interviewing systems are that: 


1. Interviewers are automatically provided with numbers to 
call. 


2. The results of all calls are automatically recorded. 


3. Callbacks are automatically scheduled. 


The first feature -- automatic provision of numbers to call -- 
potentially reduces errors in transcription. 


The callback feature can be programmed to accommodate the 
various types of non-contact. For diallings where there is no 
answer or a busy signal, the number can be regenerated 
automatically, any number of predetermined times, at program- 
med time-intervals. When a contact is made but the specified 
respondent is unavailable, the interviewer can manually key in an 
arranged callback time and any specific instruction (for example: 
Ask for Mr. Smith). This callback will be generated automatically 
at the appropriate time and with the appropriate instructions. 
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Three most frequently raised questions are: 


1. What about open-ended questions? 


No. Interviewers do not have to be able to type. Responses 
to open-ended questions still can be recorded by traditional 
paper and pencil means and entered subsequently. 


All this leads to greater efficiency and accuracy in that, again, 
it removes, from the interviewer, the task of keeping track of the 
outcome of diallings and of scheduled callbacks. 


As the record of calls is retrievable at any point during the 
progress of a study, tighter control is also possible. This is 
particularly true for samples where complicated quotas are 
involved. Computer control means that the interviewer can be 
told immediately, via the CRT, whether the respondent is 
qualified and, if so, for what cell. Less interviews are wasted and 
short cells can be identified and filled. 


Records of calls are also available by interviewer, which 
enables close monitoring of productivity by individual. 


Everything | have talked about, so far, has been concerned 
with the positive aspects of computer-assisted interviewing. 
What about problems and concerns? 


Problems can arise if the interviewer forgets to record the 
serial number, for the respondent, alongside the response. 
This makes matching-up impossible. 


2: What if the computer goes down? 


| have heard that, with some systems, a computer malfunc- 
tion can result in a loss of that day’s (or evening’s) work. 
This is obviously less than ideal and, with many systems, 
is not the case. More commonly, it seems, only the inter- 
views in progress, at the time of malfunction, are in 
jeopardy. 


3. What about interviewer training and attitude? 


With any change or innovation, there are always some 
people who cannot or will not adapt. CRT interviewing is no 
different. 


However, many interviewers enjoy the new systems and 
find them easier than the traditional methods. Also, over the 
years, as new generations ( who have been brought up in 
the electronic age) come along, this interviewing form will 
become as traditional as paper and pencil. 


The way that existing interviewers are introduced to, and 
trained on, the new systems is obviously extremely impor- 
tant also. 


Future Developments 


Computer-assisted interviewing, | think, will spread widely 
over the next few years--firstly in association with telephone 
interviewing, followed by introduction in shopping centres (again, 
a central location). 


Currently, the system cannot be applied to door-to-door 
interviewing. But who knows what the next generation of techno- 
logy will bring! 
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DESIGN AND USE OF SURVEY RESEARCH FROM 


PLANNING TO EVALUATION - SPECIAL 
CONSIDERATIONS 
Madhu Joshi, 


Program Evaluation Directorate, 
Health and Welfare Canada, 
Ottawa, Ontario. 


There are basically three ways of collecting data, namely, 
observation, questioning and simulation. 


The observation method is generally used in experiments in 
the natural sciences. However, it is also used in collecting 
information on social or economic activities where an indepen- 
dent observer can collect needed data without interfering in the 
activities being observed. Studies undertaken to obtain traffic 
counts or to determine shopping patterns in supermarkets often 
involve the collection of information through observation. 


Questioning is a direct method of getting information from 
those who have it. Administrative forms, interviews, survey 
questionnaires, all use the questioning method. By and large, 
this is the most widely used method for collecting data in the 
social sciences. 


Simulation is not a method of data collection, per se. However, 
when a reliable model of the activities under study is available, 
the simulation method can be used to generate data for a variety 
of purposes. This method is commonly used in planning. 


This presentation takes a critical look at different aspects of 
survey taking when it comes to planning, program administration 
and evaluation. Special considerations in the design, conduct 
and analysis of surveys for these three purposes and some 
current issues and developments underway are touched upon 
with a hope to generate some discussion. The material dis- 
cussed here is applicable to both private and public sector 
surveys. 


Special Considerations 


Planning in Government often involves universal or national 
surveys. The purpose of these surveys could be to determine the 
incidence or occurence of certain phenomena or to identify the 
number and distribution of individuals with certain specified 
characteristics. 


Research surveys to determine potential market for a given 
produce/service or to determine the need for a specific social 
program are some examples of such surveys. 


However, on some occasions there is a need to obtain 
information from a selected segment of the population, e.g., the 
elderly, families with pre-school children. The information is used 
to plan a social or other program for the specific segment of the 
population. Constructing a sampling frame for such surveys is 
costly and/or time consuming. Very often a special questionnai- 
re related to the specific target groups is included in a general 
survey to eliminate the need for constructing a special frame. 


Surveys conducted to monitor the operational aspects of a 
program are usually short and to the point. They need to provide 
information which can be used in operational decision-making by 
program managers. Timeliness and ease of analysis are two key 
requirements for such surveys. For some long range programs, 
tracking surveys are used to monitor the effect of the program. 
Examples of tracking studies include: periodic surveys to deter- 
mine the effect of the national energy program on energy use, 
periodic surveys to determine the effectiveness of media cam- 
paigns on the public awareness of ‘‘drinking and driving’ 
problems. 


Evaluation surveys have a multitude of problems of their own. 
The maor ones among these are: 


1. Effect of changes in the eligibility of individuals to receive 
benefits of the program. 


2. Changes in the characteristics of the buyers of specific 
products - sometimes resulting from changes in 
preferences. 


3. Changes in the size of the population receiving benefits of 
specific programs. Beneficiaries of UIC is a good example 
of such a population. 


4. Identification of individuals who belong to the population. 
The longer the time since an individual had a contact with 
the program, the more difficult it is to identify and track the 
individual. 


It is easier to identify and contact (for a survey) people who 
own a specific new model snowblower than those who 
own a specific 5-year old model. 


5. Recall of what happened a long time ago. Again, the 
longer the time elapsed between the contact with the 
program and the surveys, the more serious could be the 
problem. 


6. Very often respondents are in contact with a program or 
company on several occasions, either for the same produ- 
ce/service or for different products/services. Surveys 
enquiring the opinions and/or attitudes of these indivi- 
duals regarding a product/service may be biased by their 
experience related to other products/services. 


7. Very often, evaluation involves comparison between those 
who have received a specific benefit and those who have 
not. Constructing sampling frames for the target popula- 
tion and the controls could be a major problem. This is 
especially true when the individual may have received the 
benefit some considerable time ago, or when the program 
iS universal in nature. 


8. Some programs are designed to provide varying degrees 
of benefits and/or to provide benefits over varying periods 
of time - either continuously or intermittently. Social 
welfare programs and unemployment insurance benefits 
are prime examples of such programs. 


Collection of information to evaluate the effectiveness of 
such a program poses another set of problems. 


9. Surveys for evaluation have less freedom in the choice of 
concepts if the required concepts have been defined in a 
planning study. 
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10. Motivation of respondents - in particular those respon- 
dents who have received benefits or a service a long time 
ago - may not be very high. This could result in high 
nonresponse rates. 


This is not an exhaustive list of problems but only those | have 
encountered in my dealings with surveys for evaluation. 


In addition to these specific problems, there are a number of 
problems of a general nature. The costs of survey taking are 
increasing. Resources needed to conduct surveys are becoming 
more and more scarce. The demand for information is increasing 
faster because of the advancements in information technology. 
Consequently, the number of surveys has increased signifi- 
cantly, leading to an increased response burden. 


Public awareness and sensitivity towards rights to privacy and 
the increasing potential for unauthorized use of personal infor- 
mation, coupled with increasing response burden, have led to 
greater nonresponse (both total and partial). 


Developments Underway 


Fortunately, survey research is also entering a new phase. 
There are a number of developments underway. Among these 


are the use of new technology and statistical research in dealing 
with nonresponse. 


Random Digit Dialing and Computer Assisted Telephone 
Interviewing are designed to reduce the costs of survey taking. 
There is a potential for the use of two-way cable systems for 
collecting diary or panel type surveys. This would decrease costs 
and improve accuracy of reporting. 


Ongoing statistical research include the exploitation of admi- 
nistrative information for statistical purposes, the use of record 
linkage to develop more data thus eliminating the need to collect 
data again, insights into the motivation of respondents through 
incentives and psychological appeal, ways to deal with item and 
partial nonresponse through more efficient estimation and 
weighting and the use of multiple-frame surveys. 


These technological and statistical developments are expec- 
ted to lead to the improvement in efficiency and effectiveness of 
survey research. 


Summary 


Surveys for planning, program administration and evaluation 
pose special problems. They are also costly and time consu- 
ming. Therefore, surveys should only be conducted when there 
is an absolute need for new information. 
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DATA ANALYSIS AND STATISTICS 


Chuck Chakrapani, 
Applied Marketing Statistics, 
Toronto, Ontario 


There is a great deal of confusion, it seems to me, between 
data analysis and statistics. The confusion is so acute that many 
people equate data analysis with statistics. A cursory glance at 
most published papers and research reports reveal that conclu- 
sions are often drawn exclusively on the basis of statistically 
significant differences. In addition, statistical measures like 
averages, correlations, standard deviations get reported faith- 
fully to the fourth decimal place; multivariate analyses are 
performed without an understanding of the correlation matrix; 
large differences are ignored because they are not statistically 
significant and so on. While there is a large number of books and 
papers on statistics, there is hardly any published work on data 
analysis - more importantly on what it is not. 


Based on my experience as a professional researcher and 
data analyst, | would like to submit that: 


1. Statistics is an important part of data analysis. Yet its 
contribution is trivial in translating data into usable 
information. 


2. In survey research, calculations are done quantitatively, 
but the results are used qualitatively. Failure to under- 
stand this results in unproductive and misleading 
conclusions. 


Having made such seemingly outrageous statements, | expect 
that | owe you some explanations. 


1. Statistical measures are important, but their 
contribution to data analysis is trivial. 


This statement seems to contradict itself. Let me therefore 
explain it through an analogy: It is important for a person to 
breathe, but the fact that a person breathes in itself does not 
contribute anything towards understanding that person. In fact, 
its contribution is trivial. Statistical measures like significance 
tests likewise are important and yet trivial. A statistically signifi- 
cant conclusion is the beginning, not the end, of data analysis. 


In many cases, statistical conclusions are not even the 
beginning of data analysis. Consider the following examples: 


Example 1 

Montreal Toronto Difference 
True’ % of Users 37% 44% 7% 
Obtained Users 37% 44% 7% 
Sample Size 100 100 Not Significant 
Example 2 

Montreal Toronto Difference 
‘True’ % of Users 42% 45% 3% 
Obtained Users 42% 45% 3% 
Sample Size 1000 1000 Significant 


Significant differences, being a function of the sample size, 
often lead to conclusions that may not match reality in the 
context of data analysis. In the first example, we conclude that 
there is no difference between Montreal and Toronto while there 
is not only a large difference, but it is faithfully reflected in our 
survey. In the second example, we conclude that there is a 
significant difference while from a marketing point of view the 
difference may be insignificant. 


Since statistical significance is not necessarily related to 
marketing significance, the data analyst should answer at least 
two questions before attempting to carry out significance tests: 


1. If the obtained differences are indeed statistically signifi- 
cant, could it make any difference to my marketing 
strategy? i.e. would | consider the difference significant 
from a marketing point of view? (If the answer is ‘no’, 
significance tests are meaningless.) 


2. If the obtained differences are large from a marketing 
point of view, can | attribute the non-significant conclu- 
sions to the sample size? (If the answer is ’yes’ to this 
question, then statistical significance testing becomes 
largely irrelevant.) 


There are also instances where the use of statistics is not only 
irrelevant, but positively misleading. Consider, for example, the 
following results: 


Ratings of a New Detergent 


Toronto Montreal 
Convenient Soe Onz 
Economical 4.0 4.2 
Safe 4.3 4.3 
Cleans Well 4.2 335 
Softens Clothes 3.8 3.6 
Makes Clothes Bright Si 5.4 
Removes Stains 4.3 3.2 
Average 4.0 4.3 


From the table, it looks as though Montreal residents rated the 
product more favourably than the Toronto residents. But a closer 
look at the table tells an entirely different story. All Toronto 
averages are 4.0 + .3, i.e. it is quite likely that Toronto 
consumers did not have any stable image of the product but 
simply rated the product as an average one, no matter what the 
attribute was under consideration. If, indeed, this were the case, 
then all inter-comparisons between Toronto and Montreal are 
meaningless. Yet a statistician would have told you that on 
certain specific attributes Toronto and Montreal consumer 
ratings were significantly different. 


2: in survey research, calculations are done 
quantitatively, but the results are used qualita- 
tively. Failure to understand this results in 
unproductive and misleading conclusions. 


Statistical measures like standard errors, correlations, etc. are 
computed the same way whether it is a problem in physics or in 
market research. But the way the results are used can be very 
different. For example, if a ball-bearing has to be manufactured 
with a diameter of .0036‘' with a standard error of .0001, 
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anything beyond these specifications is unacceptable. Statistics 
and data analysis coincide here. Precision is an important part of 
the data analysis and everything depends on it. Similariy, a 
.001% difference in calculations may determine whether an 
astronaut will land on the moon or will be lost in space. 


However, corresponding statistics are interpreted differently in 
survey research. There are essentially two reasons for this: 


1. It is not possible to obtain the same precision in survey 
research because of practical considerations; and 


2. More importantly, even if such precision is possible to 
obtain, it is useless and could be misleading from a 


pragmatic point of view. 


For example, it is highly improbable that a different conclusion 
will be arrived at by the fact that 38% of the consumers like your 
product as opposed to, say 36%. If the correlation between two 
variables is .81, the conclusion drawn is not likely to be any 
different than if the correlation is .84. Therefore, high precision 
figures like r = .8344 become meaningless in survey research. In 
addition, they could also be misleading. For example, consider 
the following table: 


TABLE 1. Correlations Among Twelve Product Attributes 
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The patterns in the correlation matrix are not obvious. One 
could go ahead and perform a factor analysis on this. Doing so 
does indeed result in different factors that may be interpreted in 


.02250 
. 00497 
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.09422 
. 00000 
.02222 


. 15800 
.06185 
. 10322 
.09527 


0.00763 0.04294 -0.04760 0.22470 -0.22749 
-0.02682 -0.02401 -0.13962 -0.09211 0.13487 
-0.20327 -0.12472 -0.19723 -0.03566 0.16144 

0.20419 0.00755 0.01257 -0.09832 -0.13855 

0.00266 -0.00086 -0.23673 0.09939 0.18951 
=O2022225 50: 15800 0.06185 1.10322 0.09527 

1.00000 0.16705 0.11872 0.02953 -0.03452 

0.16705 1.00000 0.20020 0.24826 0.01620 

0.11872 0.20020 1.00000 0.04885 -0.04185 

0.02953 0.24826 0.04885 1.00000 -0.11911 
-0.03452 0.01620 -0.04185 -0.11911 1.00000 


marketing terms. Yet the same matrix, rounded to single digits, 
looks like this: 


TABLE 2. Correlations Among Product Attributes 


AB 
AB Adds body to hair = 
LM _ Leaves hair manageable =| 
LS Leaves hair soft = 
VR Gives hair styling versatility 3 
LR Gives long-lasting results 0 
MU Is mild to use 0 
HL Leaves hair healthier looking 0) 
MD Causes minimum hair damage 0 
SH __Straightens hair well 0 
SF Safe for color treated hair 2 
MI Causes minimum skin irritation -2 
EC Leaves wet hair easy to comb 0 
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Even a cursory glance at this matrix would indicate that the 
correlations are far too weak to perform factor analysis on. High 
precision reporting in this case not only obscured the patterns in 
the data but actually led to an inappropriate analysis. Whenever 
the analyst is tempted to add ‘precision’ to the numbers, it may 
be a good idea to answer the following question: 


“If | replace the last few digits with random numbers, de 
make any difference to the interpretation of the findings? 


If the answer is ’no’ - in most survey research this is in fact the 
case - then we are not dealing with relevant interpretable 
information but with noise-dominated data. 


A survey research analyst may be able to get more out of the 
data and avoid misleading interpretation if he or she constantly 
is aware of the fact that in survey research, data are quantitative 
but interpretation is qualitative. 


In such cases, which are rules rather than exceptions, statisti- 
cal tools and sophisticated techniques are nothing more and 
nothing less than alchemy. Perhaps better results may be 
achieved by simply tossing a coin. At least in this case, you can 
be right fifty percent of the time. 


It is said of Humpty Dumpty that ‘‘All the King’s horses and all 
the King’s men could not put Humpty together again’. Data 
analysis, once devoid of proper orientation is just a broken 
Humpty Dumpty. Not all our statistics, sophisticated techniques 
and significance tests can put in back together. 
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DATA ANALYSIS - IMPLICIT IN THE SURVEY 
DESIGN 


Ruth M. Heron, 

Transportation Development Centre, 
Transport Canada, 

Montreal, Quebec. 


On preparing a survey, the good market researcher first 
ensures that the questions of interest are clearly articulated and, 
then, digs into a well-stocked bag of methodological tools for the 
one, or more in combination, which can provide a framework 
within which precise answers to the questions can be generated. 
The selected statistical or mathematical model then dictates (i) 
the variables for which data are to be collected, (ii) the strategy 
for collecting these data, (iii) the form in which the data are to be 
expressed, and (iv) the analyses to be carried out on them. 
Clearly, the analyses to be used for survey data should be 
specified beforehand as parts of an inclusive model for answer- 
ing the research questions posed. 


When this approach is not taken, serious problems of data 
analysis arise. The data analyst’s headache is to be brought in 
at the end of a survey and confronted with a plethora of variables 
which were thrown into the survey pot without any regard at all 
as to how they might later be analyzed. The inexperienced might 
try to submit large numbers of such variables to one-by-one 
crosstabular analyses or to a series of t tests. Along with tending 
to produce 'significant’ relationships which are really due only to 
chance, both these approaches ignore redundancies and inter- 
actions among variables and, hence, run high risk of producing 
an entirely misleading picture of actual results. 


Hoping to overcome these difficulties and, thereby, obtain a 
holistic view of affairs, the analyst may attempt a multivariate 
procedure. However, when applied a posteriori, the selected 
model may run into data with unsuitable numerical properties or, 
in other ways, be unable to subsume all the variables for which 
data were collected. In any case, these techniques tend to 
capitalize on chance when used in this way. On balance, then, 
there is simply no satisfactory alternative to preselecting the 
types of data analyses to be used. 


More specifically, a first principle of survey data analyses is 
that those applied should be implicit in the research design. As 
such, they will encompass all the variables for which data are 
collected as a formalized framework for obtaining a comprehen- 
sive set of outcomes which contain precise answers to all 
research questions addressed by the survey. A welcome bonus 
to proceeding in this way is that the simplest and most economic 
survey design is ensured. 


An Example 


A hypothetical example will serve to illustrate this principle. 
Suppose that we have a product we wish to market, say, a 
packaged frozen food such as noodles in sauce, which can vary 
dichotomously on, for simplicity, only three features as follows: 
(1) absence or presence of additives, (2) easy or difficult 
preparation (eg. throw a pouch into boiling water vs thaw, 
arrange, and bake), and (3) gourmet or ordinary content (eg. 


sour cream with cheese, chives, and dill vs just milk and 
cheese). Combining these features factorially generates the 
product matrix below. 


No Additives Additives 

Easy Hard Easy Hard 
Gourmet A B C D 
Ordinary E ie G H 


Let us say also that a priori information is to the effect that 
such products tend to be perceived by shoppers in terms of 
various attributes, say, the six following bipolar ones: dull/lively, 
displeasing/pleasing, bad/good, plain/fancy, costly/cheap, 
heavy/light. Then a decision as to how to package this product 
for certain target populations will be made on the basis of how 
the various product features and feature interactions are percei- 
ved with respect to these attributes by the respective 
populations. 


How shall we approach this research problem? It is only with 
a set of product spaces, one for each population of interest, that 
we would be able to see at a glance how each differs in its 
perception of the eight products. There are numerous ways of 
developing product spaces, but it would be desirable to have one 
which allows us to add structure to the space by inserting feature 
and feature interaction vectors, since these fectors would point 
up detail which may not be readily discernible without them. 
More than this, we shall want to determine which, and to what 
extent, feature interactions contribute meaningfully to percep- 
tions, and to build an appropriate space from this information. 
Finally, we should like to be able to construct spaces from the 
data of given individuals, since this property would enable us to 
determine the extent of variability within populations with respect 
to Cue responsiveness. 


The paradigm having all these requisites is one recently 
proposed by Holbrook dnd Moore (1982)(1). It entails generation 
of product spaces from canonical correlation analysis (CCA) of 
three types - namely, products-based (PB), features-based (FB), 
and interactions-based (IB) - all of which use the same set of 
data. To obtain these data, our survey instrument need only 
describe the eight products and ask respondents to rate each on 
each of six scales formed from the respective six bipolar 
attributes mentioned earlier. The latter comprise a Rating Set of 
variables for all three CCAs which then differ only in how the 
product information is coded for the second set of variables. In 
the PB type, the second set is defined by seven (n-1) dummy 
variables representing the eight products. The FB type uses as 
many dummies as there are features (in our case, three, F1, F2, 
F3), a product being given a score of 1 on agiven dummy if it has 
the feature in question and a 0 otherwise. The IB type uses all 


(1) Holbrook, M. B., & Moore, W. L. Using canonical correla- 
tion to construct product spaces for objects with known 
feature structures. Journal of Marketing Research. 
1982, 19(2), 87-98. 
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the feature dummies as well as any interaction dummies of 
interest; in the latter case, a product is given a score of 1 ona 
given dummy if it has all the features contained in the interaction 
and a 0 otherwise (eg. product B would have a score of 1 on F13 
since it is both of the non-additive and gourmet type, but a 0 on 
F12 since, although it has no additives, it is not easy to 
prepare). 


A product space can be derived from each type of analysis 
provided it generates at least two pairs of canonical variates; the 
first and second Rating Variates then form the axes for a 
two-dimensional space. Interpretive information regarding a 
given axis may be obtained by correlating it with each of the 
bipolar scales in turn. Insertion of feature or feature interaction 
vectors is accomplished by multiple regression in which the 
dummy in question acts as the criterion variable and the subject 
product co-ordinates as the predictors. Multiple regression is 
used also to assess the fit of a given model; in this case, the 
subject product co-ordinates of a given axis serve as the crterion 
and the feature and feature-interaction dummies as the predic- 
tors. As the PB type contains any interactions which may be 
operating, the configuration derived from it will be the same as 
that obtained from the FB model if, and only if, there are no 
significant interactions. Thus the feature-interaction dummies 
must be added to the regression procedure one at a time so as 
to permit assessment of which are are important to the model. 
Finally, unlike the PB model, the FB and IB types permit spaces 
to be generated from the data of one individual, a property 
extremely important for assessing the extent and nature of 
variability within groups. 

To gain an idea of how this paradigm begins to answer our 
research questions, let us suppose that we have generated a 
product space for one population from the PB CCA, added 
interpretive information to the axes, and inserted the triple 
interaction vector, F123. A hypothetical outcome appears in 
Figure 1. It is clear from this albeit idealized display that the 
vertical axis divides the products into gourmet and ordinary, 
while the horizontal one divides them into additives and no 
additives. However this information alone tells us very little about 
how respondents were perceiving the products. Fortunately 
though, we find that the vertical axis correlates highly with the 
dull/lively, plain/fancy, and cheap/costly rating scales, while the 
horizontal axis correlates highly with the bad/good, displeasing- 
/pleasing, and heavy/light scales: these facts give us some 
insight into the meaning of the two dimensions. Further defini- 
tional structure is given the space by the insertion of the F123 
vector; as it is very close to the additives/no additives axis, we 
could infer that its meaning is somewhat similar, that it is 
somewhat of a bad/good dimension with displeasing/ pleasing 
and heavy/light connotations. 


Gourmet 
(lively, fancy, costly) 


F 123 
\ 


ii, No Additives 
\ \ (good, pleasing, 
rf ‘ si F light) 


Additives 


Ordinary 


A hypothetical product space with interpretive information 
for axes and F 123 vector inserted. 


The presence of the F123 vector serves to clarify the nature 
of the triple interaction. To grasp its meaning, we might first 
consider the two way-interaction, according to which the tend- 
ency to perceive ordinary products as better than gourmet 
products is greater for products with no additives. This informa- 
tion is contained in the fact that the sum of the differences 
between F and B and between E and A on the F123 vector is 
greater than the sum of the differences between G and C and 
between H and D. Our hypothetical population may find food with 
additives so undesirable that the preference for ordinary over 
gourmet food is virtually irrelevant. 


Now the three-way interaction may be discerned by observing 
that the difference between E and A is greater than that between 
F and B. This outcome tells us that the greater preference for 
ordinary over gourmet products when the products are without 
additives is greater when the product is easy to prepare. Perhaps 
a gourmet product does not seem to be so if it is easy to 
prepare! 


All things considered, the F123 vector points up that, as a 
group, these undoubtedly health-minded respondents may also 
be imbued with Protestant work ethic, since their greatest 
preference is for an ordinary food which has no additives and is 
hard to prepare. Given that tests of fit support this model, and 
that examination of individual spaces reveals within-group 
consistency, the decision as to how to package the product for 
purchase by this population is straightforward. 


Final Remarks 


Because of their ability to effect statistical control, to make 
economic summary of large numbers of inter-related variables 
and, hence, to deal comprehensively with survey research 
questions of many types, use of the more sophisticated multiva- 
riate statistical models of the kind referred to above is strongly 
advocated by this writer. Not, however, without this caveat 
which can be stated as the second principle of data analysis: 
Know both your model and your data! Knowledge of the model 
comes with study and experimentation. Knowledge of your data 
is gained only by examining the distributions, the intercorrelation 
matrix, and the variance-covariance matrix, and by trying to flush 
out anomalies at every analytical step. But this matter, about 
which a book could be written, is beyond present scope. The 
purpose of this paper will have been served if it has underscored 
the first principle of data analysis, namely, that prespecification 
of a set of analytical techniques as a means of dealing with 
survey research questions is necessary not only to ensure the 
simplest and most economic survey design, but also, and most 
importantly, to provide an inclusive and incisive approach to the 
collected data. 


Note: A more detailed paper on this subject is available from the 
author. 
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EVALUATING GOVERNMENT PROGRAMS 


Jim Brophy 

Deputy Comptroller General, 

Office of the Comptroller General of Canada, 
Ottawa, Ontario 


Abstract 


This paper describes the development of program evaluation 
in the Federal Government, and the basic principles under which 
evaluations are conducted. It also outlines the steps in conduct- 
ing an evaluation in order to ensure that it is sharply-focused and 
cost-effective. 
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This morning | would like to talk about something that is: 
perhaps somewhat different in scope from the presentations that 
have been made in the last few days; different in the sense that 


| would like to concentrate my remarks on the overall process 
and environment under which program evaluation is conducted 
in the Government of Canada as opposed to specific applications 
of program evaluation to particular fields of interest. It would not 
be my intention to dwell on the specific research designs or 
techniques of evaluation. However, if there are any particular 
questions in this regard, | will ask Dr. John Mayne of my office 
to speak to them as he is the principal advisor within the 
Comptroller General’s Office in regard to the development and 
use of evaluation methodology. 


PROGRAM EVALUATION 


TB 77-47 


EVALUATION OF PROGRAMS 
BY DEPARTMENTS AND AGENCIES 


POLICY DIRECTION 


“Departments and agencies of the federal government 
will periodically review their programs to evaluate their 
effectiveness in meeting their objectives and the 
efficiency with which they are being administered.” 


The Basic Mandate for Evaluation 


The mandate or policy under which program evaluation in the 
federal government is conducted (Treasury Board Policy Circular 
No. 1977-47) calls for all departments and agencies of the 
government to evaluation periodically all their programs to 
determine their effectiveness and efficiency. A key aspect in the 
government’s policy is found in the emphasis on the word ‘‘all’’. 
Obviously, the evaluation of some programs will prove to be 
more difficult than others; however, the bottom-line of my Branch 
is not ‘‘whether’’ a program should or can be evaluated, but 
“how’’. This is a somewhat heroic statement particularly when 
you consider the uneven evolution that the program evaluation 
function has experienced in the past and the consequences this 
has had on the current environment in which Directors of the 
function are now operating. It is, nonetheless, the goal that | and 
my Branch are striving to achieve. 


BACKGROUND 


Races SHUDY A975 


DEPT’L EVALUATION STUDY 1976 


AUDITOR GENERAL'S ACT 1977 


TB POLICY 1977-47 


An Historical Context for Evaluation in Canada 


Before actually going into a description of the current ap- 
proach to evaluation, it might be useful to spend a few moments 
in reviewing the history of the function in the last number of 
years. 


In the early half of the ’70’s, there was quite a big push to 
establish an evaluation capacity within the government. This 
development was spearheaded by Dr. Douglas Hartle who, at 
the time, was the head of the Planning Branch of the Treasury 
Board Secretariat. Over the first four or five years, about 3500 
person-years were allocated to planning and evaluation func- 
tions across the federal government, but while a number of 
departments had established dedicated program evaluation 
units, Cabinet-focused evaluations were conducted by central 
agency analysts in the Planning Branch. 
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Despite this rather considerable investment of resources, 
there was increasing doubt that departmental functions were 
delivering useful evaluation products. Two consecutive studies 
conducted in the mid-1970’s basically confirmed this. Moreover, 
these studies pointed to the fact that there was no commonly- 
held definition of evaluation either at the practitioner or the 
Deputy Minister level, and that evaluation approaches ranged 
anywhere from internal audit to general research. Perhaps most 
surprising of all, certainly to me, was that most Deputy Ministers 
really did not think that it was their responsibility to evaiuate their 
department's programs. They saw themselves as administrators 
who handed out money or delivered certain services and they felt 
that questions related to whether or not a program should 
continue to exist were basically beyond their role as public 
servants to address. 


It was around this time that the work of Mr. James Macdonell, 
the Auditor General, became particularly important to the deve- 
lopment of the program evaluation function. In 1977, the Auditor 
General received an expanded mandate from the government to 
do ‘‘comprehensive auditing’’ as opposed to what essentially 
had been a financial audit role. Part of this mandate required him 
to comment on the systems and procedures that departments 
had in place to measure the effectiveness of their programs. 
Around this time, the Auditor General’s reports were pointing to 
the fact that there was basically little effectiveness evaluation 
being done in the government and that what little was being done 
was nota part of any systematic attempt to review all departmen- 
tal programs periodically. More than anything else, the AG 
reports sent a clear message to the government that something 
had to be done. | believe it is fair to say that the Treasury Board 
policy on program evaluation that was adopted in 1977 was a 
direct response to the Auditor General’s challenge that the 
government did not know what their programs were trying to 
achieve or whether the objectives of these programs were 
actually being met. 


QUESTIONS ANSWERED BY PROGRAM 
EVALUATION STUDIES 


IS THE PROGRAM STILL RELEVANT AND 
PLAUSIBLE IN TERMS OF: 


Y ITS MANDATE, 
Y PRESENT GOVERNMENT OBJECTIVES, 
W PRESENT EXTERNAL ENVIRONMENT? 


WHAT HAVE BEEN THE EFFECTS OF THE 


PROGRAM? 


HOW WELL HAS THE PROGRAM ACHIEVED ITS 
OBJECTIVES? 


ARE THERE BETTER WAYS OF ACHIEVING THE 
PROGRAMS RESULTS? 


YW ALTERNATIVE PROGRAMS 


VY CHANGES IN PROGRAM DESIGN AND ITS 
DELIVERY 


The Types of Issues/Questions that Evaluation 
Should Address 


There are a number of dimensions to consider in describing 
to you the approach we are currently taking to program evalua- 
tion in the federal government. | would like to start by outlining 
the types of issues/questions that we would expect most 
evaluation studies would address. 


The first issue or set of questions is pretty basic and that is: 
To what extent is the current mandate of the program still 
relevant in terms of present government objectives, the external 
environment, etc.? There are a lot of programs that have been 
approved over the years that were probably good ideas at the 
time. But situations change, as you know, and some programs 
ray have outlived their usefulness, may have already met their 
original objectives or, perhaps, need to be rethought in terms of 
the demands of the current environment. Therefore, one aspect 
of a program evaluation study would be to determine whether, 
or to what extent, the original rationale of the program is still 
sound. 


Additional issues/ questions that are examined in the context 
of a program evaluation relate to the effectiveness of a program 
(i.e., the extent to which it has achieved its stated objectives) 
and, more generally, to the positive or negative impacts that a 
program may have had quite apart from this. | guess that these 
latter two types of issues constitute the most difficult aspect of 
what we are trying to do in evaluation. The problem, and | am 
sure most of you are familiar with it, is in establishing and being 
able to isolate a causal relationship between a specific interven- 
tion (A) and a specific result (X); and, it is in this area that a lot 
of time is spent on developing appropriate methodological 
techniques. By way of background, | might point out that the 
Auditor General is concerned specifically, almost exclusively, 
with the ‘‘objectives achievement’’ aspect of program evaluation 
studies. This is in keeping with his ‘‘watch-dog’’ role on behalf 
of Parliament (i.e., the Auditor General’s client) which has 
approved the expenditure of funds in the pursuit of specific 
program objectives. 


The fourth type of issue that a program evaluation would 
address relates to whether there are alternative, better ways to 
achieve the intended results of a program. | suspect that, if you 
did a survey of all the evaluation studies done to date, you would 
probably find that it is on this issue that most studies have 
focused. The explanation for this lies in the fact that this type of 
“program improvement” evaluation tends to be less threatening 
in that it generally does not lead to persons displaced out of jobs 
or to significant departmental budget cuts. Most often it entails 
general improvements in program design and delivery and the 
like. 


In summary, therefore, the types of issues/questions that 
evaluations should address relate to: the continuing relevance of 
the program; the extent to which program objectives have been 
achieved; other impacts that the program may have had; and, 
finally, better ways of achieving program results. 
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BASIC PRINCIPLES FOLLOWED 


PROGRAM EVALUATION TO BE PART 
OF MANAGEMENT IN GOVERNMENT 


V SEEN AS AN AID TO DECISION-MAKING 


V SEEN AS A MEANS OF REDUCING 
UNCERTAINTY 


A WELL-DEFINED CLIENT WHO CAN ACT 


APPROPRIATELY DELINEATED PROGRAMS 


PROCEDURES TO ENSURE: 


VW TIMELINESS OF FINDINGS 

W RELEVANCE OF SPECIFIC QUESTIONS 
V OBJECTIVITY, CREDIBILITY 

W ADEQUATE REPORTING 


A REALISTIC APPROACH TO METHODOLOGY 


A VISIBLE COMMITMENT FROM THE CENTER 


The Basic Principles Under Which Program Evalua- 
tion Operates 


| would now like to spend a few moments in outlining some of 
the basic principles which underlie the program evaluation 
function as we know it in the Government of Canada. 


The first and foremost point is that program evaluation is not 
meant to be research-oriented. This is not an off-handed criti- 
cism of research but simply an affirmation that the usefulness of 
program evaluation is directly tied to its ability to provide 
cost-effective answers to specific questions that have been 
posed by senior decision-makers within a department with 
respect to a specific program. Essentially, what Directors of 
Evaluation are being asked to do is to assist decision-makers in 
their respective departments by providing sharply-focused infor- 
mation that will eliminate uncertainty as to what individual 
programs are, or are not, achieving. 


In order to achieve this, of course, you need a well-defined 
client who, once he/she receives this information, can act on 
that information. This brings me to the second basic principle 
embodied in the federal government’s policy and guidelines on 
program evaluation; that is, that the Deputy Minister is the sole 
client for program evaluation activity within his/her department 
and, as such, is responsible for establishing the context for 
evaluation studies and for acting upon study findings. This 
statement of accountability is directly counter to the scenario | 
described earlier in my presentation where, as late as in 1976, 
Deputy Ministers did not really think it was their responsibility to 
know what their department’s programs were, or were not, 
achieving. 


A third basic principle is that all departments must have a plan 
for the periodic, cyclical review of all their programs. Tne 


establishment and maintenance of such a plan is by no means 
an easy task. Firstly, there is the problem of defining exactly what 
program units (or components as we call them) will form the 
basis of the departmental plan. When the Program Evaluation 
Branch of the Comptroller General's Office first started operation 
in 1978/79, a task force was formed using seconded departmen- 
tal personnel to look at the program structure of 16 major 
government organizations. One look at the Main Estimates (or 
Blue Book) of the government was all it took to convince 
ourselves that the breakdown of ‘‘programs”’ in it were far too 
broad and that their objectives were little more than ‘‘mothe- 
rhood’’. Therefore, what this task force had to do then, and what 
departments must do now, is to break down these broad 
program structures into program ‘‘components’’ which, while 
still having corporate significance, have clear objectives that are 
discreetly resourced and are amenable to evaluation. To give 
you an idea of the immensity of this task across government, we 
have currently broken down approximately 56 major programs of 
government into about 3500 program components. A second 
problem encountered in developing an evaluation plan is in 
disciplining individual evaluation initiatives to the priorities and 
decision-making processes of the department and the govern- 
ment as a whole. In doing so, you have to make intelligent 
tradeoffs between timeliness, reliability and credibility. There is 
simply no sense in spending all kinds of time, money and effori 
in coming up with the definitive product only to find that the 
decision point against which the study findings should have been 
focused has long since passed. This lack of a managerial 
orientation to evaluation is exactly the problem being encounte- 
red in the United States according to officials of the U.S. General 
Accounting Office with whom | have spoken. Here in Canada, we 
are committed to the notion that program evaluation will work 
only if it serves management decision-making and, in this 
context, the development of comprehensive evaluation plans by 
departments is fundamental. 


A fourth principle of program evaluation (actually it is a set of 
principles), relates to how evaluations are to be conducted; that 
is, they should be done in a manner which is objective, they 
should produce reasonably reliable evidence, and the process 
and results should be seen as credible. Objectivity is protected 
by having evaluations conducted by persons independent of the 
line manager who, understandably, has a vested interest in the 
results of a performance assessment of his/her program. Most 
departments have a dedicated Program Evaluation Group re- 
porting to the Deputy Minister either directly or through a steff 
Assistant Deputy Minister. The successful conduct of an evalua- 
tion depends not only on its objectivity, but also on its credibil- 
ity. In large part, this credibility comes from the confidence that 
line managers have in the objectivity of the evaluator, in the 
evaluator’s knowledge of what he/she is doing, and in the 
openness of the evaluation process. So it is very important that 
evaluation studies be credible, that the Director of evaluation be 
credible, and that line managers be ‘‘involved’”’ in the evaluation 
process by having input into terms of reference for studies, and 
in reviewing/commenting upon the evaluation reports. This 
emphasis on credibility from a managerial perspective is a clear 
departure from the type of evaluations done in the early '70’s 
when the Treasury Board did the evaluations themselves. At that 
time, you saw some very objective and independent evaluations. 
But one of the major reasons why | think none of the evaluations 
resulted in substantive changes to programs was that the line 
managers were not involved. What that meant was that when the 
Planning Branch came up with the results of a program evalua- 
tion study, the line manager would say: ‘‘Well that’s a great 
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evaluation, and | agree that according to your statistics, etc. the 
program has not met its objectives but, those weren't the 
objectives of the program. You guys just didn’t understand what 
that program was trying to achieve.’’ | mentioned reliability. 
Assuming for the moment that perfect reliability is possible, it is 
quite frankly not affordable, particularly in today’s environment of 
restraint. What we are looking for are practical evaluation 
approaches that produce credible, useful, timely and cost- 
effective information that will give a Deputy Minister a ‘‘warm 
feeling in the pit of his stomach’ that he can make a decision 
with a reasonable degree of confidence. 


The last principle upon which the current approach to program 
evaluation is founded, and the most important, is that it must 
have solid and visible commitment from the Deputy Minister 
since it is he/she who establishes the tempo of the department 
and directs the corporate decision-making process. One of the 
prerequisites for evaluation, and indeed it is in place in most 
departments, is that the Deputy Minister establish a departmen- 
tal policy which describes how evaluation will be done in the 
department and who will play what roles with respect to its 
operation. A key aspect of this policy is the role that the Deputy 
Minister defines for himself/herself in determining which pro- 
grams are going to be evaluated when and what questions are 
to be asked. What we have found, and not surprisingly, is that 
when the Deputy Minister is most involved, the function is most 
successful. If you do not have this visible and active involvement 
of the Deputy Minister, what you often have is an evaluation 
group just going through the motions and trying to satisfy, if you 
will, the Comptroller General's Office that there is an evaluation 
function in place. So the most important principle has to be that 
there be a visible commitment from the Deputy Minister that 
he/she wants evaluation to work. 


PROGRAM EVALUATION PROCESS 


@ PRE-EVALUATION PLANNING 
(EVALUATION ASSESSMENT) 


@ ANALYSIS AND REPORTING 
(EVALUATION STUDY) 


© DECISION MAKING AND 
IMPLEMENTATION 


Steps in the Evaluation Process 


| have spoken of the history and the principles that underlie 
our current approach to program evaluation. | would now like to 
concentrate my remarks on the actual evaluation process. 


When we first started doing evaluations we did not emphasize 
the first aspect, i.e., the need to plan the evaluation study, and 
what we found was that as a result a great deal of money was 
being wasted. We charged into doing a lot of evaluation studies 
and when we got half-way into them, we found out that the data 
were not there, that the methodology was inappropriate, or that 
the evaluation lacked focus in terms of the questions that it was 
addressing. Essentially, what we saw in operation was what | 
would term the ‘‘Hoover Vacuum Concept’. Evaluators were 
charging in and sucking up all the information that was around, 
collecting all kinds of data, spending all kinds of money in the 
hope that, at some point when you emptied out that bag and 
sifted through all this information, you would come up with 
something useful. To avoid this type of inefficiency and ineffecti- 
veness, what we are doing now is to plan better evaluation 
studies through a process we call ‘‘evaluation assessment’. 


Basically, what the evaluation assessment seeks to do is to 
determine the questions you want to and can have answers to, 
the kinds of information that are available, the types of methodo- 
logies that could be used, and so forth. Actually, we seem to 
have been almost too successful in emphasizing this stage of the 
process; so much so that, quite frankly, there is almost an 
element of overkill. We are finding that, generally, there is too 
much time spent on this front-end planning, and that, for some, 
the evaluation assessment is becoming an end in itself as 
opposed to the means whereby better and more sharply-focused 
evaluation studies can be subsequently done. To counter this, 
we are backing up a little by advising departments of the danger 
of spending non-productive time on this front-end stage, when all 
that is really necessary is to satisfy yourself that you can 
eventually conduct a study and to determine the general lines of 
investigation that the study will incorporate. 


The seond stage of the evaluation process is the evaluation 
study itself. Again, the key themes here are to be realistic and 
practical and to produce cost-effective and useful information. 
However, it is not enough simply to have a function in place that 
produces evaluation reports, and this brings me to the third stage 
of the evaluation process - implementation and follow-up. 


Evaluations are only as useful as the follow-up processes to 
them. First of all, the line manager has to be brought on side; 
he/she has to understand what the study did or did not do and 
what it is recommending. These processes also have to make 
allowance for recommendations to be considered by the Deputy 
Minister, for decisions to be taken, and for subsequent action to 
take place and be monitored. 
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Program Evaluation: A Retrospective and Prospec- 
tive Viewpoint 


| hope that the above remarks have given you an idea of what 
program evaluation, in the current governmental context, is 
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trying to accomplish. In closing, what | would like to do now is to 
give you a flavour of the pace of change that has occurred in the 


past and my feeling as to what we can expect to see in the near 
future. 


The Program Evaluation Branch of the Comptroller General's 
Office has been in operation for two years of which | have been 
in the Branch for most of this time. When the Branch first started 
to monitor the activity of corporate-based evaluation units in 
government departments, there was only one evaluation study 
underway. Now we are looking at approximately 130 studies 
which relate to programs that cover off expenditures of about 13 
billion dollars. This is a great deal of progress in a rather short 
period of time, although | have no illusions that all of these 
studies are, or will be, of good quality. Indeed, it seems likely to 
me that a number of these first studies will have to be redone at 
some point in time. 


Even in retrospect, | know of no other way of getting the 
function ‘‘up and running’. It is very difficult to go out and hire 
people that have ready-made skills in program evaluation. The 
only way is to recruit the best people you can and have them 
learn by doing. We are finding that, in a number of cases, it takes 
two to three studies before you get a credible, high quality 
product that can be used and acted on by the Deputy Minister. 


It is going to take time. We are probably three to four years 
away from the point that we can really say that the vast majority 
of evaluation studies being done are good quality products. But 
we are getting there; and, as you can see, there is a fair amount 
of initiative and desire on the part of many people within the 
federal government to see that this function works and that it 
becomes an integral aspect of the overall management 
process. 


Thank you. 
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ASSESSING THE QUALITY OF THE RESEARCH 
PROJECT 


F. C. Commins, 
Research Management Group, 
Toronto, Ontario. 


Abstract 


This paper presents the concept that quality in and satisfaction 
with a research project cannot be ensured merely by carrying out 
the prescribed number of interviews or applying the correct 
techniques. There are additional requirements such as experi- 
ence, intuition, and a careful blending of a number of other 
components, including the cooperation of the client, to produce 
“The right amount of surprise’. 
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‘You might wake up some morning to the sound of something 
moving past your window in the wind. And if you’re quick enough 
to rise you'll catch the fleeting glimpse of someone’s fading 
shadow...” 


(Lind, 1966) 


Good morning, everyone. It’s my responsibility to talk to you 
for the next thirty minutes on the subject of evaluating marketing 
research projects. 


As you're probably aware, this Conference was originally 
scheduled to take place last Fall. Consequently, I’ve had over a 
year to think about my topic, and to discuss it with quite a few 
of my friends. On hearing the title of my paper, almost all of them 
suddenly remembered conflicting engagements for this week. 
I’m not sure how you feel, but ‘‘Assessing the Quality of the 
Research Project’ didn’t immediately strike me as being one of 
the more exciting issues of the day either. 


What I’m going to try to do, however, is to look at the question 
from what | hope is a fresh perspective. 


Why do we want to consider this particular subject today? 
We've just heard Jim brophy of the Comptroller General’s Office 
talk about evaluating Government programs (Brophy, 1982), and 
this same basic theme has been or will be addressed by several 
other Conference speakers. 


If ‘‘evaluation’’ involves survey research of some sort - as it 
surely does - how is this work itself to be judged? 


How, then, should we do this? In struggling over this speech, 


| kept coming back to one image... ‘‘the bad study” and ‘‘the 
good study’’. 


That wasn't That was 
a good 
study! 


too good 
a study! 


In essence, this seemed to sum up what | was trying to 
capture: the distinction between the outcome on the left versus 
the outcome on the right. 


Moreover, in staring at these two characters, several other 
thoughts occurred to me as well. For instance, ‘‘Who are those 


guys, anyway?” 


1. That is, who should be responsible for making this ‘‘good 
study’’/‘‘bad study’’ assessment? 


2. Secondly, when should the project be assessed? Presuma- 
bly, after it’s been completed, but what are the implications 
of this timing? 

3. Thirdly, on what criteria should an assessment be 
made...? 


4. And, finally, what on earth can we possibly mean by 
“quality’’? 


These are the issues that I’d like to try and address. 


First of all, the question of who makes the assessment... In 
any research project, there are at least two principal players: the 
client and the researcher. And each of these two sees marketing 
research in a different context. 


Or) 


Researcher 
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To the client, it’s a tool; to the researcher, it’s a craft. In this 
regard, marketing research is a lot like the creative process in 
advertising; from the client’s point of view, this, too, is a tool, 
intended to fulfill a specific objective. The copywriter - or the 
researcher - accepts this need for a functional tool, yet has, in 
addition, a desire to have the design aesthetically and intellec- 
tually satisfying in its own right. 


In consequence, if we return to my original image of the ‘‘good 
study’”’ and the ‘‘bad study’, it’s evident that there can be 
conflicting assessments from the two ‘‘evaluators’’. 


Client Researcher Client Researcher 


2 


3 


Researcher Client Researcher 


Client 


Outcome number one, in the northwest quadrant, is the result 
we're all striving for, while outcome number four, in the south- 
east, has, at least, the dubious advantage of consistency. But 
what of the other two scenarios, where we have contrasting 
opinions? Number two, in the northeast, is a rare occurence, | 
would hope, because it would virtually constitute fraud: a study 
that the client believes is a valid one, but that the researcher 
knows is on shaky ground. The result in the southwest corner is 
more common, I’m afraid: the ‘‘good’’ study that the client 
doesn't like for some reason. 


And, in my opinion, these studies in Region Three must be 
regarded as failures, because | believe the client casts the 
deciding vote on ‘‘quality’’. I’ll develop this point further a little 
later. 


Now for my second question... When should the assessment 
of ‘research quality’ be made? In practice, it's a two-stage 
process. In accepting the research proposal, the client is pre- 
sumably making some statements about his perception of 
“research quality’ - particularly if competitive proposals have 
been involved. Yet, in the end, it’s probably only after the 
completion of the project that the final assessment of this elusive 
“quality’’ can be made. 


This raises the issue of the nature of the research contract. 
Does the researcher simply agree to go through a certain 
process - to conduct a specified number of interviews, to 
tabulate them, to report them - or does he promise something 
more? This question is brought into sharpest focus for me in the 
case of qualitative research. Has the research contract been 
satisfactorily fulfilled when 32 qualified respondents have been 
recruited, when eight hours of discussion have taken place and 
a report free of typos has been produced? 


Factors such as these are necessary but not sufficient to the 
completion of a satisfactory research project. But they are 
necessary; any firm that consistently fails to fulfill the research 
specs with respect to due dates, cost estimates and such 
“details” will have difficulty maintaining client relationships no 
matter how ingenious their study designs. From the client’s 
perspective, failure to deliver the report on time is a clear, 
operational definition of ‘‘poor quality’. 


But what else is necessary? 


It’s been stated that the purpose of marketing research is to 
provide information that will assist in the decision-making 
process. Surely, then, this must be one of the most important 
criteria on which a project should be assessed. 


I’m in agreement with the views expressed by Vince Barabba 
of Eastman Kodak, the opening speaker of this Conference. 


Earlier this year, Mr. Barabba spoke of the market intelligence 
function of his employer: 


Market intelligence is held accountable to evaluate the impact 
information has had on the decision-making process. Instead of 
saying, from an evaluation point of view, 'How accurate were we 
or how precise?’ as a traditional market researcher might, we 
have the responsibility of saying, ’How quickly did the informa- 
tion get into the decision-making process? How widespread was 
its utilization? And did it affect the decision, or would we have 
made the decision with or without the information?’ In a word, did 
we make our objective to affect the decision-making process? 


(Barabba, 1982) 
To repeat, I’m in complete agreement with this position. 


Yet, surely we hope that marketing research not only affects 
the decision-making process, but affects it in the right direction. 
Here’s another relevant quotation, this from an Editorial in the 
British Journal of the Market Research Society: 


It is easy to say that research which is not implemented is 
wasted, but besides making actionable recommendations re- 
search can also perform valuable roles as a ritual - it can clarify 
issues, delay overhasty decisions, encourage group involve- 
ment, enable more risky decisions to be taken by providing a 
scapegoat and so on. 


Unfortunately the same arguments could be advanced for 
methods of divination, such as those used by the Romans. If 
research is only used as a ritual its actual degree of accuracy 
ceases to be important. Of course it is still necessary that there 
should be a superficial appearance of accuracy - no doubt 
Roman priests spent even more time discussing how close the 
birds had to be before they constituted a valid augury than we do 
in discussing the level of response needed for an acceptable 
sample. 


(JMRS, 1981) 
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This notion of ‘‘accuracy’’ relates directly to the final question 
of the four that | raised at the outset of this paper. | would like to 
suggest that we should think of the marketing research process 
as ‘‘the search for the right answer’’ - the ‘‘bright, elusive 
butterfly’’ of my theme music. 


Yet this perspective leads to the ultimate riddle of marketing 
research. 


There is a Zen koan which asks “If a man is seeking an 
unknown destination, how will he know when he has reached 
it?’’ (| heard that on an old ‘‘Hawaii Five-o”’ rerun.) 


We undertake survey research because there’s something we 
don’t know. Given that premise, how do we recognize ‘‘the right 
answer’ when we have found it? 


One difficulty is that occasionally when we go searching for 
right answers we find wrong answers - worse still, we don’t 
identify them as such. As a friend of mine is fond of pointing out, 
survey research always produces facts, but sometimes they’re 
“incorrect facts’’. And it’s often very hard to tell the little buggers 
apart. 


But surely the solution to this problem is obvious, you say. 
“Quality” research is what will produce ‘“‘correct facts’’ rather 
than their misleading counterparts. One should simply apply 
“good research technique’ to each study. 


But is this really the answer? A synopsis of recommended 
practices from the Statistical Activities Secretariat on Contract- 
ing Out Survey Research (Statistics Canada, 1977) indicates 
that among the principal points which should be covered in an 
evaluation of a supplier's performance is ‘‘the quality of the work 
performed’’. The only difficulty is that they provide no suggestion 
as to how to assess this ‘‘quality’’ - which, | suppose, is one of 
the reasons that I’m speaking here today. (In fairness, the 
document implies that one should judge the final output against 
the ‘‘bid’’, in terms of costs, timing, controls, and so on. Yet, as 
I’ve suggested earlier, surely these criteria are simply a 
minimum.) 


It was at this point that | decided that it might not be a bad idea 
to look up the meaning of ‘‘quality’’ in a dictionary. My Webster’s 
defines ‘‘quality’’ as ‘‘the degree of excellence’, an explanation 
that | didn’t find all that helpful. What it did make me realize, 
however, is that ‘‘quality’”’ is a very abstract concept, and that 
what’s missing is the element of cost. In addition to ‘‘quality’’, 
one of the other principal points which Statistics Canada sug- 
gests should be covered in an evaluation of research should be 
“‘an assessment of value received for the expenditure’’. In fact, 
on consideration, | felt much more comfortable with ‘‘value’’ as 
the idea we were pursuing, and so | promptly revised the title of 
my paper, which now addressed the problem of assessing the 
value of the research project. 


The same Webster’s defines ‘‘value”’ as ‘‘the relative worth, 
utility or importance”’ of an item, and | think this is a much more 
relevant term. If marketing research is to make a worthwhile 
contribution to the decision-making process, it must do so at a 
cost that is not prohibitive. 


Therefore, | would argue that ‘‘quality’’ or ‘‘value’’ in market- 
ing research is achieved when we have as our objective: to find 
“the right answer’’ at the lowest possible cost. 


This is precisely the view held by the President of the Treasury 
Board. Not long ago, Donald Johnston was quoted as saying: 


... Fhrough program evaluation, information is available both 
on questions on whether the right programs are being funded, 
and whether results are being obtained at least cost. In this 
sense, program evaluation provides a parallel to the role of the 
bottom line in the private sector. 


(Johnston, 1982) 


If this perspective of ‘‘least cost’’ is appropriate for program 
evaluation, surely it should be extended to the process by which 
those evaluations are made. 


This emphasis on ‘‘value’’ or cost is particularly relevant here 
in Canada. As I’m sure | don’t have to point out to this audience, 
the accuracy of a given sample size is not related to the size of 
the universe from which it’s drawn. 


In consequence, if 2,000 interviews are required for a given 
study in the US, 2,000 interviews would be necessary in Canada 
as well. 


But the cost of the Canadian study - generally about the same 
as that in the US - would have to come out of an overall program 
budget that is normally only one-tenth as large. 


In Canada, in particular, therefore, it’s often necessary to try 
to substitute ingenuity for money. The best solution is not to 
always rely on textbook definitions of ‘“‘correct technique’ and 
large sample sizes. 


In advertising, it’s been said that television networks don’t 
charge any more to run good commercials than they do to run 
poor ones. Similarly, research firms don’t charge any more to 
ask clever questions than they do to ask dumb ones. 


Unless we can get our friends on Parliament Hill to repeal the 
laws of statistics, we’re always going to be under pressure to 
come up with efficient research designs in Canada. The relatively 
high cost of conducting accurate research in this country can be 
a constraint, but it can also be a beneficial challenge. It helps 
break the mind set that the solution to any problem is to throw 
money at it. Just as in poetry, for example, where the rules of the 
sonnet can be a stimulating discipline, so too are we forced to be 
creative if we must come up with an effective research design 
with a cost ceiling of say, $10,000. In fact, given the very limited 
research budgets we sometimes have to work with, a better 
poetic analogy than the sonnet might be the haiku. 


In sum, to achieve this ‘‘quality’’, this ‘‘value’’, I’m suggesting 
that we must not only find ‘‘the right answer’, but we must do 
so without over-spending on methodology. 


lf, in order to address a given research problem, we recom- 
mend a study design that involves 4,000 personal interviews and 
three callbacks, we must feel that this will give us ‘“‘the right 
answer’. Yet, from the point of view of management's decision- 
making process, would we probably not have come to the same 
“right answer’ if we had made only two callbacks? If we had 
completed only 3,000 interviews? Only 2,000 interviews? 


It is the duty of the researcher, | believe, to trim the research 
design down as far as possible without compromising the 
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integrity of the study. This is a narrow wire to walk indeed, and 
| feel judgement and experience are all we really have to guide 
us. (For better or for worse, I’ve never known anyone who 
routinely applied Bayesian analysis to attempt to assess the net 
value of various research options.) 


In thinking about this process of paring the research design 
down to its essentials, I’m reminded of a game we played on the 
beach as children. Sand was piled up and a stick placed on top. 
Each player had to remove some of the sand until one finally 
went too far and made the castle collapse. 


This is what we must do with the research design, but here the 
castle is invisible, the collapse unseen. How far can we go in 
holding the cost down to a reasonable level without turning 
“right answers’ into ‘‘wrong answers’’? Yet, to repeat, the 
solution is not to do 4,000 interviews on every study. 


An editorial by Douglas Banik in the Journal of Advertising 
Research addressed this question of the search for the right 
answer. 


The concept of research itself is premised upon the assump- 
tion that we, as researchers, pursue the truth, even when the 
truth hurts. That’s why integrity is the researcher’s chief commo- 
dity... Can we really know truth? Of course not, no more than we 
can know the Platonic ideal. But we can get closer and closer ..... 
The fact is that when researchers give up pursuit of the truth, 
they give up their reason for being... 


(Banik, 1981) 


So that’s what ‘‘quality’’ comes down to, for me: the truth - 
which is simply another way of saying ‘‘the right answer’. 


And ‘‘value’’ is that ‘“‘right answer’ achieved at the lowest 
possible cost. How do we know for sure that we’ve found it? We 
don't, or we wouldn't have been conducting the study. 


That wasn't That was 
too good 
a study! 


a good 
Study! 


To return to one of my opening slides, | would suggest that we 
achieve the condition on the right - the ‘‘good study’’ - when two 
things happen: one objective, the other subjective. 


For the objective criterion, | would argue for the ‘‘market 
intelligence’ perspective of Eastman Kodak: Did the research 
actually influence the decision-making process? 


For the subjective criterion, | would argue for a shared sense 
by the researcher and the client that they have learned some- 
thing, that they truly understand the situation under investigation 
more than they did before conducting the research - and that as 
little money as possible was wasted in acquiring that learning. 
(This theme was developed very well a few years ago by John 
Coulson in an article on ‘‘how to Reduce Research Waste’’.) 


(Coulson, 1977) 


This sense of understanding is very subjective, but in my 
opinion it’s the way most good practitioners and users of 
marketing research actually judge studies. The goal is always to 
achieve a new way of looking at the subject - which is also what 
I’ve tried to accomplish in this paper. William Wells once said that 
a good study contains just ‘‘the right amount of surprise’, (Wells, 
1975) and | can’t think of a better phrase. 


That ‘‘good study”’ feeling, that sense that ‘‘the right answer’ 
has been found can come from quantitative or qualitative 
research, so | don’t think the solution rests in sample size or in 
confidence intervals. The confidence must come from within, 
from experience, intuition and from techniques applied intelli- 
gently within the framework of a reasonable budget. 


John May has suggested, again in the British Journal, 
that... 


The --- subject which more writers should address themselves 
to is what research is, rather than what it purports to be. The gap 
needs to be bridged between the grand logical ‘‘scientific 
method’’ definitions that abound in the text books and give the 
impression of ‘‘precision’’, ‘‘exactness”’ and ‘‘certainty’’, and the 
rather messy, intuitive and inexact practice that actually takes 
place. 


(May, 1981) 


You'll have noticed, I’m sure, that I’ve carefully avoided 
talking about field validation, keypunch verification and other 
interesting stuff like that - things that most of the literature would 
equate with ‘‘quality’”’ in marketing research. We all know these 
sorts of steps are necessary, but again they’re not sufficient in 
themselves to ensure ‘‘the right answer’’. Moreover, it’s not 
problems in these areas that produce what clients perceive to be 
‘bad studies’, in practice. 


Researcher 
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To return to another earlier slide, by my criteria this outcome 
would have to be assessed as a ‘‘bad study’’. The client is the 
umpire. In those instances where he feels unhappy with a study 
that the researcher thought was good, the fault most often lies 
with the report. If there is one single area where | think that 
marketing research can and should strive for an absolute - rather 
than a relative - improvement in ‘‘quality’’, it’s in the standard of 
the reporting and presentation. In this instance, it’s really true 
that ‘‘the mdium is the message’’. 


| think of marketing research as a craft, even though | know 
management must use it as a tool. And, as with any craft, both 
the appeal and the frustration lies in the fact that, no matter how 
well you do it, it's never as good as it could be. 


Yet what we should strive for, | believe, is the perspective of 
a North York shoemaker named Pete, quoted in an article on 
how Canadians feel abut their work. He said ‘‘The idea | go on 
is that | can fix shoes better this week than | could last week.”’ 
| think if we can hold onto this attitude to our own work, the 
‘‘quality’’ will take care of itself. 


For me, the enjoyment and satisfaction | find in marketing 
research lies precisely in this search for the right answer, 
coupled with the knowledge that, in the end, we really have 
nothing but our intuition to tell us when we’ve found it. 


And that’s the thought I'd like to leave with you. 
Don’t be concerned, it will not harm you. It’s only me pursuing 


something I’m not sure of. Across my dreams, with nets of 
wonder, | chase the bright, elusive butterfly of love. 


(Lind, 1966) 


May you all find your butterflies. Thank you. 
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EVALUATING COMMUNICATIONS 


Peter Hume, 
Sigma Communications Research, 
Toronto, Ontario. 


Abstract 


This paper describes a widely-accepted set of principles for 
evaluating communications. It argues that advertisements which 
play on the emotions of readers/viewers are more effective than 
those which give people logical facts, and that pictorial images 
are more effective than text in conveying a message. 


**¢ os ee 


Ladies and Gentlemen: 


The first part of my talk is for the you that likes to sleep nights, 
to keep your job and so on. The second part of the talk is for the 
you that wants to live dangerously. 


The first part of my talk has been enthusiastically endorsed by 
twenty-two of the leading advertising agencies in the United 
States, the second part is still being debated within my own 
agency. (Actually, that’s not fair; there is growing acceptance of 
much of what it stands for, especially among our creatives who 
have to live dangerously every day, but also like to sleep 
nights. ) 


The duality of the talk is an attempt to address the central 
conflict of the marketer's job; the balancing of risk and responsi- 
bility. We researchers have got to help resolve that conflict. 


Let us try to make this all a little less abstract. Let’s assume 
we all work for a company that wants to break into the coffee 
business; that it has a fine Brazilian blend that taste testing tells 
us is competitive with the leading brands. Without detailing the 
total strategy, let us assume that we need to make considerable 
impact with our advertising to encourage the trial that should 
lead to conversion and success. 


How would you set about evaluating a testimonial campaign, 
set in Rio, using a beautiful Brazilian woman and her suave, 
worldly beau, Ronald Biggs, the Great Train Robber who no 
longer worries about a job, certainly doesn’t sleep nights, and 
who sips a lot of good coffee beside the pool of the Copacabana 
Palace? 


Twenty-two leading US agencies come to our rescue. These 
agencies have agreed on the nine PACT (Positioning, Adverti- 
sing, Copy Testing) principles of good copy research which | 
shall now detail in the interests of risk minimization and a coffee 
brand to build a dream on. 


I’ve rearranged the order of the PACT principles to give them 
what seems to me to be better logic. At greater risk, I’ve also 
reworded them to help us better understand what they mean. 


The first PACT principle says that before putting the Ronald 
Biggs commercial through its paces, we should have some 


model in our minds of how advertising works. If you are Chuck 
Chakrapani, your model may be something like Howard and 
Sheth’s, complex and with arrows flying in all directions. Or it 
may merely hold that if advertising is to work it must (i) capture 
attention; (ii) say something relevant about the brand; (iii) and do 
this with sufficient interest and charm to (iv) influence the 
likelihood of purchasing our brand of coffee. 


The second PACT principle says that any good test should be 
relevant to the specific objectives of the campaign. If a key 
objective of our Rio campaign is to create a buccaneering brand 
personality for the coffee, or to lead coffee afficionados to notice 
a particular nuance of its flavour, we had better find some way 
of building these into the test, such as brand personification or 
a taste test or even a product placement, after the advertising 
has been exposed. 


The third PACT principle says that we must decide in advance 
(Doug and Bob McKenzie would say a priori) what our action 
standards will be. Lever Brothers around the world hold that if 
any negative in an advertising test exceeds 15% it must be 
rectified. This standard may not apply if you’re researching a 
campaign to explain the validity of Inco’s operation in Sudbury. 


Like marriage vows, action standards should be agreed upon 
up front and meticulously observed. This is easier said than 
done, especially with multiple measures of effectiveness. 


The fourth PACT principle holds that the test should reflect the 
subtleties of the target group that the advertising must reach out 
to an influence. If our coffee brand has been positioned against 
those users of the existing popular brands who feel that they are 
boringly domestic and lack mystique, we’d better find (disguised) 
ways of qualifying such people for the sample. 


| suggest that we should always try to go beyond demograph- 
ics to behavioural or better still, personality dimensions in the 
sample selection. 


The fifth PACT principle simply says that the more finished the 
material the better the test (and that if two pieces of advertising 
are being tested against each other we should try to ensure that 
they are both equally finished, or if you prefer, both equally 
unfinished). This is no great revelation until we remember that 
different degrees of finish effect different aspects of the test. 
Mary Jane Schlinger and my ex-colleague, Leila Green, who is 
Head of Creative Research at Leo Burnett in Chicago, have 
shown that while animatics are an adequate test of a commer- 
Cial’s capacity to communicate, they are a poor test of its ability 
to touch people emotionally. 


The sixth PACT principle holds that any good test will provide 
a number of different measures of effectiveness, rather than one 
single ‘‘magic number’’. With tests like Schwerin, and in un- 
skilled hands, Day-after-recall (DAR) too often acted as tow 
trucks for fine advertising which simply was not designed to work 
like Tide or Ivory’s. (Procter and Gamble have repeatedly used 
DAR as part of their evaluation procedure for many years but one 
can only believe that they use more than the ‘‘related recall’’ 
magic number.) 


Joe Plummer of Young & Rubicam has argued not just for a 
multiplicity of measures in one test but a multiplicity of different 
tests, on the grounds that no one test is sensitive to all that we 
are trying to assess. 


a7 oie 


Our agency joined with J. Walter Thompson and Warner 
Lambert to sponsor a Queen’s University study into what is 
known about the factors that influence DAR. We've learned a 
great deal about what it can and cannot do for our clients’ 
commercials, and how to read the results it gives. We would 
refer you to Dr. Steve Arnold at Queen's for further details. 


As to the measures (of effectiveness) we should include and, 
remembering our earlier model of advertising, we may want to 
measure 


Impact 
Communication 
Empathy 
Persuasion 


with some diagnostic questions to help pinpoint the reasons for 
a good or bad performance on each aspect of the test. 


The next PACT principle says that we should ask ourselves 
whether multiple exposures of the material are important. 


Herb Krugman in a celebrated piece of work entitled: ‘‘How 
much is enough?” argued that there are three psychologically 
significant responses to any new commercial. He says people 
respond with: 


1. What is it? 
2. What of it? 
3. What about one? 


So we should attempt to arrange for three showings of any 
new commercial - though this must be done in a way which does 
does not seem forced. 


Perhaps we should also attempt to simulate the total cam- 
paign - television, magazines and radio, where these apply. 


The second-to-last PACT principle argues for controls in good 
copy testing - the use of experimental procedures such as 
matched samples to help control for the background ‘‘noise”’ 
that can easily influence the results more than advertising 
differences. (Though this is difficult to acknowledge, advertising, 
which is our obsession, is no big deal to consumers, and often 
what we are dealing with is wisps of meaning and feeling as a 
result of seeing the material, particularly in the short-term.) 


There are many other kinds of control, alternating the order of 
exposure of two commercials, but the practicalities of the real 
world often make big samples and accurate matching difficult. 
These difficulties throw many marketers back on norms, the 
averages of many other people’s tests, not the ideal given the 
uniqueness of what one is often attempting to do. 


Another important control in Canada is the including of several 
geographic regions in any important test. My introduction to this 
was to follow the testing by Goldfarb of the Unity Office Constitu- 
tion Campaign from the East Coast across to Vancouver. | 
couldn’t believe how differently it was received in each city. A 
British researcher said he felt that the regional differences in a 
test we did in Quebec, Ontario and Alberta were due to an 
aberration in the way the electronic ratings were administered. 
Not so. The diagnostics told us that they were due to regional 
differences in headspace. The test was for a major oil 
company. 


Finally, the nineth of the PACT principles relates to the 
reliability and validity of the testing procedure. In simple terms, 


reliability is the extent to which repeated testing of the same 
material will give substantially the same results. The so-called 
test/retest reliability measure. This is a cornerstone requirement 
often neglected by test constructors. DAR proved to be rather 
unreliable but at least it has been subjected to some of this kind 
of scrutiny. 


Validity is the measure of how well the test measures what it 
says it does, and how relevant this is to what the advertising has 
to achieve. 


Shirley Young’s attack on DAR centred on the fact that she 
could find little evidence that the DAR scores were related to the 
commercial’s performance in the marketplace. It is the most 
neglected criterion in advertising testing and certainly contrib- 
utes to the rather meagre understanding we have of how 
advertising really works. 


The ideal measure of validity would be the extent to which the 
test results can be used to predict sales of the brand in the 
marketplace. After all, advertising is, in the final analysis, 
intended to ring the cash register. But all manner of nonadvertis- 
ing factors enter into the equation. Which means that we must 
use more limited validation studies, like checking on the extent 
to which results from one test correspond with another. 


These principles will not be new to you but are difficult to put 
into practice. However, if we fail to observe them, we are in 
danger of turning copy testing (and ultimately, advertising) into 
daylight robbery. 


Which brings me to the part of the paper that will appeal to that 
part of you that wants to live dangerously! 


The topic of this half is evaluating the emotional response to 
advertising, which is the source of much of its real power. I’m 
going to argue that much of this comes from subtle nuances, 
little twists, that we too often fail to measure adequately. Bill 
Bernbach, when asked about storyboard testing in Paris last 
year said: ‘‘How do you storyboard a smile when the sweet 
sadness of that smile might be the whole campaign?”’ 


Given that many of you are from Ottawa, I'd like to use an 
example, possibly apocryphal, from the world of politics to 
demonstrate that a more indirect approach to communications 
can work more powerfully than a more direct approach. 


The example relates to Giscard D’Estagne campaigning to 
become the Premier of France. Given his politics, personal 
wealth and aristocratic style, the opposition parties decided they 
would attack him on the grounds of being elitist and out of touch 
with the real needs of France. This was said over and over again 
by the opposition candidates, the leftist media, etc., yet the polls 
moved steadily in his favour. 


Close to the election, he agreed to a televised ‘‘Meet the 
Press’’ session at which the opposition media railed away at him 
again. One young woman representing a left-of-centre newspa- 
per said nothing until close to the end of the session when she 
said: ‘‘Monsieur, I'd like to ask just one question. What is the 
price of a Metro ticket?’’ Giscard reportedly looked startled and 
then guessed a price that had not applied for ten years. His polls 
dipped and he almost lost the election. The questioner was using 
the same strategy but the little twist made a world of difference 
in terms of emotional response. 
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Richard Nixon’s early campaigning was hampered less by the 
many articles about how ambitious and expedient he was than 
by the oblique line: ‘‘Would you buy a used car from this 
man?” 


In the last Ontario elections, a province that has known better 
times, the Bill Davis government responded to Stuart Smith’s 
direct attack on the Conservatives’ poor management of the 
economy and the number of young people who were being 
forced to look elsewhere for jobs, not by explaining anything, but 
with an advertising jingle that urged people to ‘‘Keep the 
promise’’, whatever that meant. 


The Progressive Conservatives in Ontario appear to favour 
inference and emotional suggestion, even in times when people 
are indeed losing sieep worrying about their jobs. The ‘‘Ontario, 
Preserve it, Conserve it’’ advertising, using idyllic, soft focus 
shots of the province’s scenery, is ostensibly about the environ- 
ment, but many believe that it ‘‘rubs off’’ subtly on the govern- 
ment of the day. By contract, much Federal Government adver- 
tising seems more explicit, is possibly more responsible, but may 
not match the emotional power of the more oblique campaigns 
we have discussed. 


Even in these stressful times when consumers and voters are 
being more tough-minded, it may still be true to say that it’s not 
what a politician says, but how he looks saying it, that 
matters. 


Which leads into a discussion of the different ways in which 
words and pictures work. There is a lot of evidence that the left 
side of the brain is concerned with our ability to use words and 
the right side of the brain is involved with our processing of 
picture images, with music, and with the emotional side of our 
makeup. 


In our testing of communications, we find that pictures and 
music have an immediacy of feeling which is not matched by the 
more plodding and mundane logic of the accompanying text. 


There is even some support for this from the work done by 
McGill's great brain surgeon, Wilder Penfield, who did many 
brain operations under conditions of local anaesthesia and found 
that each time he electrically stimulated particular points on the 
cortext, the patient remembered long forgotten incidents toge- 
ther with how he/she felt at the time. Memory may file not just 
facts but feelings. 


Why do we continue to skew our communications testing to 
the verbal, (left-brain) aspects of the advertising? The PACT 
acronym betrays this bias in that CT stands for Copy Testing. 
Copy is the written, verbal part of advertising, which in an age of 
broadcast may be a minor part of the experience that is 
advertising. 


If you go back to the first part of my talk, and you are all 
well-educated left-brain types, what do you remember most 
vividly? The nine PACT principles or Ronald Biggs and the girl 
from Ipanema sipping coffee beside the pool of the Copacabana 
Palace? 


Of course, words can be made to work, with feeling. The 
words you remember from the film ‘‘Extra Terrestrial’ are, I'll 
bet, “‘E.T., phone home.” Think of the brilliantly round-about 
way Steven Spielberg makes us feel good in a recession. By 
making us cry for the plight of a loathsome little creature from 


who the hell knows where! Certainly, he understands that 
emotion is everything, and that indirectness is emotionally more 
powerful than the simple facts. 


Ronald Biggs for Corcovado Coffee would stand little chance 
of surviving most of the testing we now use. Yet he is a 
potentially powerful right-brain property. Especially if the com- 
mercial were undeniably Brazilian in character and he was filmed 
as a real person, welcoming the chance to introduce us to 
something that he has discovered there and wants us to 
share. 


Let’s look at one or two more right-brain-oriented techniques 
for each of the key stages in the development of a campaign: 


(i) Planning/Strategy Stage: While finding out about 
the key benefits of the product category and the extent to which 
our brand delivers them, let us also take time to explore the 
brand’s character - its ‘‘emotional equity’, using oblique tech- 
niques such as brand personification, lifestyle picture sorting 
with questions to help classify respondents more richly than 
demographics make possible. If you are developing a strategy to 
“explain’’ soft drinks to mothers with children, try using a 
question such as ‘‘How many glasses of soft drink do you feel 
that a 10-year-old child should have in a week?”’ Most strategies 
fail to challenge and excite our creative people. They are too 
literal. 


(ii) Creative Development Stage: In the quest for an 
idea which meets the strategic objectives and has the power to 
move people, try using the ‘‘living’’ ideas your creative people 
come up with rather than bland concept statements. Let cre- 
atives ‘‘talk’’ their ideas through in focus groups or one-on-one 
interviews or using VTR filmed ‘‘human storyboard’. We find this 
is SO much closer to the intended commercial and, though 
purists would argue that the presenter looms too large in such 
a test, we’ve found it helps creatives to work and rework the clay. 
It can give them more ‘‘feel’’ than anything else we know. 


(iii) Execution Testing Stage: At the more finished 
stage, perhaps using an animatic, a pilot commercial or a 
photomatic we try to cover the four performance factors de- 
scribed earlier in the test. We use a clutter approach to check not 
only impact, intrusiveness but to see how the commercial 
performs when people are not set to watch it 20:20. Which they 
seldom are in the real world. Most of the questioning focus is not 
on the commercial but on what it seems to be doing for the 
brand. 


We might also wish to consider the latest eye cameras, such 
as the portable Telcom models, which can show not just which 
zone a person is looking at and for how long, but which line of 
copy in the ad or which element on the television screen is being 
viewed. 


We also like the Clucas type of question in such a test: ‘‘What 
went through your mind?”’ 


(iv) Exposure Stage: With strongly emotive advertising 
we are looking to next-day replaying of a group of commercials 
using a spare cable channel, and a telephone interview which 
commercials were seen and how consumers felt about them. As 
a right-brain refinement on DAR, we welcome the work done by 
ASI in Los Angeles using a next-day replay of commercials over 
cable for ‘‘recognition’’ testing. This recognition approach 
reportedly gives 60-70% higher scores than traditional DAR 
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recall for commercials that are more dependent on the feeling- 
s/pictures than on words. Where this is not a factor, we would 
use DAR with more control than in the past. Testing within a 
feature film can make for 8-9% higher scores than if the 
commercial goes out in a low interest sitcom. 


(v) Tracking Stage: While we believe the main focus of 
tracking should be to measure the job the campaign has done for 
the brand, we would argue that the advertising questions are an 
important adjunct to this. If television is an important part of the 
campaign we like to use some visual aids, frames from the 
commercials to avoid the problems of trying to access right-brain 
impressions with left-brain recall questions. (If the telephone is 
used, try using brief descriptions of the commercials.) Ask 
people who remember the advertising to tell you how they feel 
about it to help us design more empathetic commercials in the 


future. With remote controls we are entering the era of the 
“Invited Commercial’? and while development research bears 
the greatest responsibility for this, the tracking can help too. 


What this inevitably very speculative second section adds up 
to is a profound belief that while advertising does many things, 
its most important job is making people feel good about the 
brand, government policy, the political candidate, or whatever. 


The PACT principles will help us rigorously test for this but we 
may also have to use techniques which sacrifice some rigour for 
relevance. Which help us come to terms with the subtle things 
that people hear with their inner ear and what this makes them 
feel about an obscure brand of coffee presented by a famous 
Train Robber. 
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WHAT LIES BEYOND THE HORIZON? 


Mark Lovell, Directeur general, 
Groupe Innova Inc., 
Montréal, Quebec. 


First, we should define the horizon. This is whatever seems to 
be the practical set of limitations that controls personal or 
corporate development. Recession provides limitations of a kind: 
it discourages many companies and government departments 
from spending on research on a scale that corresponds to that 
of a growth industry, an industry in which one is prepared to 
invest for the future. Other limitations are psychological, how- 
ever, not external factors. Under-investment in marketing re- 
search is a chronic problem, ensuring that the average problem 
is addressed by methods that more properly belong to an earlier 
era. The psychological reasoning behind this can be expressed 
as two basic convictions that run right through the industry: 


(i) That the trained human mind is the key element in all 
phases of a research project: any introduction of new 
equipment that involves another mind, or another kind of 
training is suspect and resented. 


(ii) That the development of experience in particular tech- 
niques is too important a matter to be jettisoned in favour 
of developing new techniques, involving new 
technology. 


These factors put the average research company (and many 
a research buyer, too) in a frame of mind that accepts innovation 
only at the level of interpretation, ensuring that the means of data 
collection and analysis remain fixed, restricted to a number of 
well-thumbed pages in a procedures manual. 


Both these limitations, the economic and the psychological, 
contrive to keep marketing research in a state where only lip 
service is paid to progress. To be ‘‘leading edge’ either in 
hardware or in software, is in reality seen as straying beyond the 
boundaries of good sense. 


There are many changes that would lie within our grasp if 
these horizons were less restrictive. In the end, the marketing 
research industry will probably be forced to come to terms with 
them, by impatient groups of research buyers. 


Most research is either strategic or reactive. It may seem to be 
time to mount a re-appraisal study of a particular market. Or 
something happens - falling sales, a ccampaign change, compe- 
titive activity, etc. - to force a re-visit, from a particular point of 
view. It is still relatively uncommon for a company to have a basic 
information system running continuously, or up-dated regularly. 
This allows market planning to be kept under constant review; 
and for accurate modelling exercises to test the likely effects of 
strategic or tactical changes. The advantages are clear. Detec- 
tion of changes in trend, or of areas which are bucking the trend 
can be immediate, and corrective action can be focussed on 
these things. Similarly, it provides a ready-made testing ground 
for any manipulation of marketing variables - whether of introdu- 
cing a new variety, doubling trade or consumer spending, or 
phasing it out. Research becomes an on-going operation, to 
which you plug in from time to time, rather than a new step that 
you take when circumstances make it seem necessary. 


The mechanisms for continuous research procedures have 
been with us for a long time. Easier, cheaper and more conve- 
nient ways of up-dating, organizing and accessing the data have 
been developing more slowly. But they are hardly new. Incorpo- 
rating them into ordinary research thinking is what takes the 
time. 


Let us consider a simple example of what | mean. Two 
companies are engaged in the same business, on the same 
scale, and with the same willingness to find out what they need 
to do in marketing, and to take the necessary action. Let us 
further assume that one of them, Company A, is cast in a more 
traditional mold, and is more conscious of the current horizons 
of marketing research than of what may lie beyond them. 


Both companies receive the same information on sales, on a 
national basis. Both companies keep the data in a memory store 
attached to a computer. But Company A locks it away in a 
department called ‘‘Sales Data’. Company B has the information 
permanently available to its executive teams, who can make use 
of these advantages: 


- Terminal at their disposal 
- Training on terminal use 


- Simple programs for accessing, re-organizing, and making 
‘What-if’ calculations 


- Up-dates on new software and its uses 
- Consultants for specific help or more complex applications 


- Links with data in other computers - e.g. Statistics Canada 
population or trade figures 


SATA = 


There is a comparison of what then happens at Company A 
and Company B. Note that the functions of marketing research 
and marketing are beginning to merge together in Company B. 
Does this sound more like a threat, or an opportunity for greater 


efficiency in decision-taking? Depending on your answer, the 
horizon is either rigidly fixed (in the case of perceived threat) or 
it is more flexible. 


Company A 
Symptom: Falling sales in Moose Jaw, reported 
by an alert regional sales manager. 
Procedures: Discussion of possible reasons with 


sales staff, research. 


Discussion of possible research to 
investigate problem. 


As problem is observed becoming 
larger, need for research increases: 
arguments about scale, location, 


Company B 


Regular read-outs indicate falling 
sales in several cities, including 
Moose Jaw. 


Sales data is examined to determine 
which variables constitute the most 
to the incidence of falling sales: 
main problems identified. 


Effects of corrective action are 
investigated by altering values 
of controllable variables in the 
territory concerned; effects of 


purpose ... 


A research proposal is prepared, 


submitted, discussed, revised 


Ad hoc research begins. 


This example could have dealt with consumer information - on 
awareness, penetration, repurchase patterns, image, etc. - just 
as much as with sales data. In fact, the same hardware, and the 
same principle of software availability can accommodate both 
types of data, and probably more besides. 


What is needed is an attitude towards investment in the way 
that one is being informed; and towards organizing the company 
thinking into sharing out was is available, rather than compart- 
mentalizing it. The pay-off is in action time, in a beter basis for 
decisions and in general preparedness vis-a-vis market 
change. 


There are important changes here in the nature of the 
research manager’s function, and in the kind of services that an 
outside supplier might usefully offer. 


The research manager at Company B can no longer keep his 
expertise in a black box, from which conclusions emerge. He is 
playing his hand of bridge with the cards face upwards on the 
table. He has to be that much more capable with the technology 
than the others, he has to anticipate deficiencies in the system 
or possible new applications before they become obvious to 
everyone else. As regards the data coming in, he has to prowl 
around, examining quality and accuracy, and he has to re- 
appraise policy on storage. All this is required of him as well as 
the ability to scrutinize the methodology of data collecting 
services to say nothing of having his own views on what the data 
implies for the brand groups. It’s a challenging assignment. 


On the supplier side, there will be two crucial things to do: 
(1) understand what is going on among the more sophiscated 


research users, and be able to compare their potential, 
and to discuss their input requirements 


different levels of corrective 
action are then compared. 


ee Decision taken. 


(2) be prepared to function more as a consultant on an 
on-going basis, helping clients to derive the maximum 
usefulness from continuous or stored data, as opposed to 
concentrating on ad hoc surveys. 


This, too, is challenging. Obviously | am talking about changes 
in emphasis and degree of involvement, rather than 180 degree 
turns. But the less adaptable researchers will simply lose 
business. 


There are plenty of other items that might be called ‘‘over the 
horizon’’ that people are still unwilling to come to terms with. In 
fact, they are here already. But since they are inconvenient, 
since they interrupt established marketing thinking, and research 
practice, they are largely ignored. 


(a) The electronic approach to data collection. This has 
begun, but it is still very minor in Canada. With the 
increase in ‘‘No-go”’ areas, and the worsening economics 
of conventional labour-intensive field forces, it must 
increase. 


(b) | Household fragmentation is here to stay. Singles are hard 
to interview, hard to ascribe to a shopping pattern. But 
they and unofficial double-households are increasing in 
importance. New ways of building them into data collec- 
tion have to be developed. 


(c) Broadcasting is becoming narrow casting. General inter- 
est media are sick. The fine-tuning of media selection 
requires a much more innovative approach in the future, 
with particular attention to special interest categories. 


The methods for many of these things exist, or could be made 
to exist quite easily. But so long as the investment and the 
psychological horizons remain in force, they will remain 
distant. 


Sich 


MARKETING RESEARCH - THE SEVEN DEADLY 
SINS 


Peter T. Zarry, Chairman, 
Advertising Management Group, 
Ottawa, Ontario 


Mr. Chairman, Ladies and Gentlemen: 


| have a feeling that you truly are happy even perhaps 
delighted to see me to-day - not for what | might say or what 
expertise | may have or how witty | may be, but rather because 
I’m the last speaker you are going to be exposed to at this 
conference. 


This conference has had a fully packed schedule, loaded with 
heavy topics, riddled with expert speakers and chock full of 
‘food for thought’. Now it’s time to relax. In the next twenty 
minutes, I’m going to offer you one man’s opinion of what | 
believe are the Seven Deadly Sins in Market Research. Actually, 
using the original seven deadly sins (pride, covetness, lust, 
anger, gluttony, envy and sloth) couched with examples in the 
market research area in itself could be an interesting speech. 
However to-day, | will propose my own list - make up your own 
list and see how they compare. 


ae First Sin - Forgetting that the role of Market research 
is not to make decisions 


This is probably the most often stated truism and at the same 
time is the most violated principle in the research area. The real 
role of market research is to gain sight in consumer behaviour 
and attitudes. It should rarely be considered as a decision 
making component in the marketing process. With all respect to 
the MR practitioners in the audience, | would even suggest that 
too much respect is given than is due to the process (‘‘it’s been 
researched”’ or ‘‘research tells us’’ are the magic blankets that 
whisk away all of our problems and concerns). 


2. Second Sin - Spending too little time between sup- 
plier and buyer to define the basic research task to be 
accomplished. 


What information is it that we really want to obtain abd how 
should we do it? This is often very time consuming requiring that 
rare commodity called disciplined thinking. More often than not, 
we revert to the ‘‘recipe’’ approach which is great for making 
brownies but hardly suffices for good research. Research is not 
a continually repetitive process. We must constantly ask what we 
need and why we need it. A great deal of research to-day tends 
to be repetitive standard fare (right from the ‘‘recipe book’’) and 
totally lacks creative input. 


3. Sin Three - Failure to avoid pre-judgement on ex- 
pected results. 


This is an extremely dangerous sin because of its penetrating 
bias which seeps into all parts of the research process. Pre- 
judgement (the sincerest form of self flattery) can diabolically 


influence problem definition, objective setting, research design 
(even tabulation design) and worst of all, influence attitudes on 
evaluation. Pre-judgement is a sport, like the NFL, which should 
go out on strike in perpetuity. Watch for pre-judgement verifica- 
tion masked in a coat of budget restrictions. 


4. Sin Four - Penalizing creativity in the tendering 
process. 


Buying research is not buying bricks. One of the most 
reprehensible crimes committed to-day in marketing research, is 
preventing the development of a great new approach to informa- 
tion gathering because of the damnable concept of lowest quote 
in order to obtain the assignment. Let’s never lose our perspecti- 
ves on percentages spent on research versus % spent in media 
(in advertising) or the cost of launching an ineffective major 
program. ‘‘Penny-wise-pound foolish”’ is all too prevalent in 
market research to-day (incidentally, the opposite can often be 
problematical also). 


5: Sin Five - Accumulating information that is not 
actionable or is redundant 


Information accumulation is expensive and not being able to 
use insights gained is sinful waste. Again, this goes back to our 
basic principle of clearly defining what we are searching for. The 
question that must be asked (and answered specifically) is ‘‘what 
are we going to do with the information once obtained?”’ And an 
unacceptable answer is that the information will be ‘interesting’. 
Further basic questions must be addressed such as ‘“‘how really 
important is increasing accuracy by larger sample size - or are 
these recommendations of larger samples simply a method for 
the analyst to escape from the basic intense analysis that must 
be done to ferret insights and trends?’’. What is the cost of this 
added information versus improved accuracy? 


6. Sin Six - Falling into the trap of becoming a purist 
statistician instead of a market research interpreter. 


Research techniques and analytical tools have come a long 
way in the last 20 years - we now have multivariate analysis, 
computer perceptual mapping etc. and | could go on. Are we 
attacking MR with the same mentality as Mt. Everest climbers 
and utilising techniques simple ‘‘because they are there’. | 
suggest that often we have a tendency to make issues much 
more complicated and complex than the consumer makes it 
because of our new research ‘‘toys’’ which are just begging to 
be used and will place the users in ‘‘avant garde’’ posture. We 
are prone to manufacture ‘‘shadings’’ that are realistically 
immaterial in the basic decision process. 


Th Sin Seven - Transmitting bad information (either 
deliberate or unintentional) and passing it off as valid 
research. 


| don’t have to spend time on the ‘‘please the client’ syndro- 
me. We all know there are many methods for market researching 
any problem and | know I'll have great difficulty in getting even 
2 of you in the audience to agree on any one particular method. 
But are we even doing a greater disservice when we force an 


2 Ge 


inappropriate technique (because of our familiarity with it or a 
vested financial interest) when a better or simpler method is 
available. Let’s stop getting complex answers to simple pro- 
blems. We are constantly exposed to the danger of assuming a 
behavioral response based on attitudinal response questionnai- 
res. The GIGO principle in computers is equally valid in market 
research - bad briefing leads to bad objective formulation leads 
to bad design leads to bad analysis. 


These are my choice for the Seven Deadly Sins in Marketing 
Research. I’m sure that you have others that you feel should be 
on the list or some that should supplant some that | have 
included. If you have any that you feel particularly strong about, 
please give me a call at 995-1427 and I'll start compiling the 
“great sin list in the sky”. 


Thank you. 
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